<meta charset='utf-8' />

<meta http-equiv="X-UA-Compatible" content="chrome=1" />

<meta name="description" content="q : q - Text as Data" />

<link rel="stylesheet" type="text/css" media="screen" href="stylesheets/stylesheet.css">

<title>Usage</title>

<script type="text/javascript" src="javascripts/google-analytics.js"></script>

<!-- HEADER -->

<div id="header_wrap" class="outer">

<header class="inner">

<select id="language_switcher" onChange="window.location.href=this.value">

<option value="index.html" selected>English</option>

<option value="ja/index.html">Japanese</option>

</select>

<img id="q-logo" src="images/q-logo1.png"/>

<a id="forkme_banner" href="https://github.com/harelba/q">View on GitHub</a>

<h1 id="project_title">q</h1>

<h2 id="project_tagline">q - Text as Data</h2>

<iframe src="http://ghbtns.com/github-btn.html?user=harelba&repo=q&type=watch&count=true&size=large" allowtransparency="true" frameborder="0" scrolling="0" width="170" height="30"></iframe>

<section id="downloads">

<a class="zip_download_link" download href="https://github.com/harelba/q/archive/1.7.1.zip">Download the last stable version zip file </a>

<a class="tar_download_link" download href="https://github.com/harelba/q/archive/1.7.1.tar.gz">Download the last stable version tar.gz file</a>

<a class="rpm_download_link" download href="https://github.com/harelba/packages-for-q/raw/master/rpms/q-text-as-data-1.7.1-1.noarch.rpm">Download the latest stable version RPM</a>

<a class="deb_download_link" download href="https://github.com/harelba/packages-for-q/raw/master/deb/q-text-as-data_1.7.1-2_all.deb">Download the latest stable version Debian package</a>

<a class="executable_download_link" title="Single file executable, for systems with python installed" href="https://cdn.rawgit.com/harelba/q/1.7.1/bin/q">Download the single-file executable</a>

<a class="windows_download_link" title="Installation for Windows systems. Adds q.exe to the path. Please send me any comments" href="https://github.com/harelba/packages-for-q/raw/master/windows/setup-q-1.7.4.exe">Download the single-file executable</a>

</section>

</header>

</div>

<div id="sidenav">

<h1 id="general">General</h1>

<ul>

<li><a href="index.html">Home</a></li>

<li><a href="install.html">Download and Install</a></li>

<li><a href="requirements.html">Requirements</a></li>

<li><a href="limitations.html">Limitations</a></li>

<li><a href="future-ideas.html">Future Ideas</a></li>

</ul>

<h1 id="documentation">Documentation</h1>

<ul>

<li><a href="usage.html">Usage</a></li>

<li><a href="examples.html">Examples</a></li>

<li><a href="tutorial.html">Tutorial for Beginners</a></li>

</ul>

<h1 id="other">Behind the Scenes</h1>

<ul>

<li><a href="changelog.html">Change Log</a></li>

<li><a href="implementation.html">Implementation</a></li>

<li><a href="rationale.html">Rationale</a></li>

</ul>

<h1 id="author">Author</h1>

<ul>

<li><a href="contact.html">Contact</a><br/>Twitter: <a href="https://twitter.com/harelba">@harelba</a><br/>Email: <a href="mailto:harelba@gmail.com">harelba@gmail.com</a></li>

</ul>

<h1 id="social">Social</h1>

<ul>

<li>Twitter: <a target="_blank" href="https://twitter.com/search?f=realtime&q=%23qtextasdata&src=typd">#qtextasdata</a></li>

</ul>

</div>

<!-- MAIN CONTENT -->

<div id="main_content_wrap" class="outer">

<section id="main_content" class="inner">

<h2>

<a name="synopsis" class="anchor" href="#synopsis"><span class="octicon octicon-link"></span></a>SYNOPSIS</h2>

<pre><code>`q &lt;flags&gt; &lt;query&gt;`

Simplest execution is `q "SELECT * FROM myfile"` which prints the entire file.

</code></pre>

<h2>

<a name="description" class="anchor" href="#description"><span class="octicon octicon-link"></span></a>DESCRIPTION</h2>

<p>q allows performing SQL-like statements on tabular text data. Its purpose is to bring SQL expressive power to the Linux command line and to provide easy access to text as actual data.</p>

<p>Query should be an SQL-like query which contains filenames instead of table names (or - for stdin). The query itself should be provided as one parameter to the tool (i.e. enclosed in quotes).</p>

<p>Use <code>-H</code> to signify that the input contains a header line. Column names will be detected automatically in that case, and can be used in the query. If this option is not provided, columns will be named cX, starting with 1 (e.g. q "SELECT c3,c8 from ...").</p>

<p>Use <code>-d</code> to specify the input delimiter.</p>

<p>Column types are auto detected by the tool, no casting is needed.</p>

<p>Please note that column names that include spaces need to be used in the query with back-ticks, as per the sqlite standard.</p>

<p>Query/Input/Output encodings are fully supported (and q tries to provide out-of-the-box usability in that area). Please use <code>-e</code>,<code>-E</code> and <code>-Q</code> to control encoding if needed.</p>

<p>All sqlite3 SQL constructs are supported, including joins across files (use an alias for each table).</p>

<p>See <a href="https://github.com/harelba/q">https://github.com/harelba/q</a> for more details.</p>

<h2>

<a name="query" class="anchor" href="#query"><span class="octicon octicon-link"></span></a>QUERY</h2>

<p>q gets one parameter - An SQL-like query. </p>

<p>Any standard SQL expression, condition (both WHERE and HAVING), GROUP BY, ORDER BY etc. are allowed.</p>

<p>JOINs are supported and Subqueries are supported in the WHERE clause, but unfortunately not in the FROM clause for now. Use table aliases when performing JOINs.</p>

<p>The SQL syntax itself is sqlite's syntax. For details look at <a href="http://www.sqlite.org/lang.html">http://www.sqlite.org/lang.html</a> or search the net for examples.</p>

<p><strong>NOTE:</strong> Full type detection is implemented, so there is no need for any casting or anything.</p>

<p><strong>NOTE2:</strong> When using the <code>-O</code> output header option, use column name aliases if you want to control the output column names. For example, <code>q -O -H "select count(*) cnt,sum(*) as mysum from -"</code> would output <code>cnt</code> and <code>mysum</code> as the output header column names.</p>

<h2>

<a name="runtime-options" class="anchor" href="#runtime-options"><span class="octicon octicon-link"></span></a>RUNTIME OPTIONS</h2>

<p>q can also get some runtime flags. The following parameters can be used, all optional:</p>

<pre><code>Options:

-h, --help show this help message and exit

-v, --version Print version

Input Data Options:

-H, --skip-header Skip header row. This has been changed from earlier

version - Only one header row is supported, and the

header row is used for column naming

-d DELIMITER, --delimiter=DELIMITER

Field delimiter. If none specified, then space is used

as the delimiter.

-t, --tab-delimited

Same as -d &lt;tab&gt;. Just a shorthand for handling

standard tab delimited file You can use $'\t' if you

want (this is how Linux expects to provide tabs in the

command line)

-e ENCODING, --encoding=ENCODING

Input file encoding. Defaults to UTF-8. set to none

for not setting any encoding - faster, but at your own

risk...

-z, --gzipped Data is gzipped. Useful for reading from stdin. For

files, .gz means automatic gunzipping

-A, --analyze-only Analyze sample input and provide information about

data types

-m MODE, --mode=MODE

Data parsing mode. fluffy, relaxed and strict. In

strict mode, the -c column-count parameter must

be supplied as well

-c COLUMN_COUNT, --column-count=COLUMN_COUNT

Specific column count when using relaxed or strict

mode

-k, --keep-leading-whitespace

Keep leading whitespace in values. Default behavior

strips leading whitespace off values, in order to

provide out-of-the-box usability for simple use cases.

If you need to preserve whitespace, use this flag.

Output Options:

-D OUTPUT_DELIMITER, --output-delimiter=OUTPUT_DELIMITER

Field delimiter for output. If none specified, then

the -d delimiter is used if present, or space if no

delimiter is specified

-T, --tab-delimited-output

Same as -D &lt;tab&gt;. Just a shorthand for outputing tab

delimited output. You can use -D $'\t' if you want.

-O, --output-header

Output header line. Output column-names are determined

from the query itself. Use column aliases in order to

set your column names in the query. For example,

'select name FirstName,value1/value2 MyCalculation

from ...'. This can be used even if there was no

header in the input.

-b, --beautify Beautify output according to actual values. Might be

slow...

-f FORMATTING, --formatting=FORMATTING

Output-level formatting, in the format X=fmt,Y=fmt

etc, where X,Y are output column numbers (e.g. 1 for

first SELECT column etc.

-E OUTPUT_ENCODING, --output-encoding=OUTPUT_ENCODING

Output encoding. Defaults to 'none', leading to

selecting the system/terminal encoding

Query Related Options:

-q QUERY_FILENAME, --query-filename=QUERY_FILENAME

Read query from the provided filename instead of the

command line, possibly using the provided query

encoding (using -Q).

-Q QUERY_ENCODING, --query-encoding=QUERY_ENCODING

query text encoding. Experimental. Please send your

feedback on this

</code></pre>

<h3>

<a name="table-names" class="anchor" href="#table-names"><span class="octicon octicon-link"></span></a>Table names</h3>

<p>The table names are the actual file names that you want to read from. Path names are allowed. Use "-" if you want to read from stdin (e.g. <code>q "SELECT * FROM -"</code>)</p>

<p>Multiple files can be concatenated by using one of both of the following ways:</p>

<ul>

<li>Separating the filenames with a + sign: <code>SELECT * FROM datafile1+datafile2+datefile3</code>.</li>

<li>Using glob matching: <code>SELECT * FROM mydata*.dat</code>

</li>

</ul><p>Files with .gz extension are considered to be gzipped and decompressed on the fly.</p>

<h3>

<a name="parsing-modes" class="anchor" href="#parsing-modes"><span class="octicon octicon-link"></span></a>Parsing Modes</h3>

<p>q supports multiple parsing modes:</p>

<ul>

<li>

<code>relaxed</code> - This is the default mode. It tries to lean towards simplicity of use. When a row doesn't contains enough columns, they'll be filled with nulls, and when there are too many, the extra values will be merged to the last column. Defining the number of expected columns in this mode is done using the <code>-c</code> parameter. If it is not provided, then the number of columns is detected automatically (In most use cases, there is no need to specify <code>-c</code>)</li>

<li>

<code>strict</code> - Strict mode is for hardcore csv/tsv parsing. Whenever a row doesn't contain the proper number of columns, processing will stop. <code>-c</code> must be provided when using this mode</li>

<li>

<code>fluffy</code> - This mode should not be used, and is just some kind of "backward compatible" parsing mode which was used by q previously. It's left as a separate parsing mode on purpose, in order to accomodate existing users. If you are such a user, please open a bug for your use case, and I'll see how I can incorporate it into the other modes. It is reasonable to say that this mode will be removed in the future.</li>

</ul><h3>

<a name="output-formatting-option" class="anchor" href="#output-formatting-option"><span class="octicon octicon-link"></span></a>Output formatting option</h3>

<p>The format of F is as a list of X=f separated by commas, where X is a column number and f is a python format:</p>

<ul>

<li>X - column number - This is the SELECTed column (or expression) number, not the one from the original table. E.g, 1 is the first SELECTed column, 3 is the third SELECTed column.</li>

<li>f - A python formatting string - See <a href="http://docs.python.org/release/2.4.4/lib/typesseq-strings.html">http://docs.python.org/release/2.4.4/lib/typesseq-strings.html</a> for details if needed.

** Example: <code>-f 3=%-10s,5=%4.3f,1=%x</code>

</li>

</ul><h2>

<a name="examples" class="anchor" href="#examples"><span class="octicon octicon-link"></span></a>EXAMPLES</h2>

<p>Example 1: <code>ls -ltrd * | q "select c1,count(1) from - group by c1"</code></p>

<pre><code>This example would print a count of each unique permission string in the current folder.

</code></pre>

<p>Example 2: <code>seq 1 1000 | q "select avg(c1),sum(c1) from -"</code></p>

<pre><code>This example would provide the average and the sum of the numbers in the range 1 to 1000

</code></pre>

<p>Example 3: <code>sudo find /tmp -ls | q "select c5,c6,sum(c7)/1024.0/1024 as total from - group by c5,c6 order by total desc"</code></p>

<pre><code>This example will output the total size in MB per user+group in the /tmp subtree

</code></pre>

<p>Example 4: <code>ps -ef | q -H "select UID,count(*) cnt from - group by UID order by cnt desc limit 3"</code></p>

<pre><code>This example will show process counts per UID, calculated from ps data. Note that the column names provided by ps are being used as column name in the query (The -H flag activates that option)

</code></pre>

<h2>

<a name="author" class="anchor" href="#author"><span class="octicon octicon-link"></span></a>AUTHOR</h2>

<p>Harel Ben-Attia (<a href="mailto:harelba@gmail.com">harelba@gmail.com</a>)</p>

<p><a href="https://twitter.com/harelba">@harelba</a> on Twitter</p>

<p>Any feedback/suggestions/complaints regarding this tool would be much appreciated. Contributions are most welcome as well, of course.</p>

<h2>

<a name="copyright" class="anchor" href="#copyright"><span class="octicon octicon-link"></span></a>COPYRIGHT</h2>

<p>Copyright (C) 2012--2014 Harel Ben Attia</p>

<p>This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3, or (at your option) any later version.</p>

<p>This program is distributed in the hope that it will be useful,but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA </p>

</section>

</div>

</body>

</html>