pythonlearn/html-270/book016.html at master · StudyCourse/pythonlearn

History

331 lines (325 loc) · 25.2 KB

Raw

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

<!DOCTYPE html>

<html>

<head>

<title>Visualizing data</title>

</head>

<body>

<hr />

<h1 class="chapter" id="sec186">Chapter&#XA0;15&#XA0;&#XA0;Visualizing data</h1>

So far we have been learning the Python language and then

learning how to use Python, the network, and databases

to manipulate data.In this chapter, we take a look at three

complete applications that bring all of these things together

to manage and visualize data. You might use these applications

as sample code to help get you started in solving a

real-world problem.Each of the applications is a ZIP file that you can download

and extract onto your computer and execute.

<h2 class="section" id="sec187">15.1&#XA0;&#XA0;Building a Google map from geocoded data</h2>

<a id="hevea_default807"></a>In this project, we are using the Google geocoding API

to clean up some user-entered geographic locations of

university names and then placing the data on a Google

map. <div class="center"><img src="book021.png" /></div>To get started, download the application from:www.py4inf.com/code/geodata.zipThe first problem to solve is that the free Google geocoding

API is rate-limited to a certain number of requests per day. If you have

a lot of data, you might need to stop and restart the lookup

process several times. So we break the problem into two

phases. <a id="hevea_default808"></a>

In the first phase we take our input &#X201C;survey&#X201D; data in the file

where.data and read it one line at a time, and retrieve the

geocoded information from Google and store it

in a database geodata.sqlite.

Before we use the geocoding API for each user-entered location,

we simply check to see if we already have the data for that

particular line of input. The database is functioning as a

local &#X201C;cache&#X201D; of our geocoding data to make sure we never ask

Google for the same data twice.You can restart the process at any time by removing the file

geodata.sqlite.Run the geoload.py program. This program will read the input

lines in where.data and for each line check to see if it is already

in the database. If we don&#X2019;t have the data for the location, it will

call the geocoding API to retrieve the data and store it in

the database.Here is a sample run after there is already some data in the

database:<pre class="verbatim">Found in database Northeastern University

Found in database University of Hong Kong, ...

Found in database Technion

Found in database Viswakarma Institute, Pune, India

Found in database UMD

Found in database Tufts University

Resolving Monash University

Retrieving http://maps.googleapis.com/maps/api/

geocode/json?sensor=false&address=Monash+University

Retrieved 2063 characters { "results" : [

{u'status': u'OK', u'results': ... }

Resolving Kokshetau Institute of Economics and Management

Retrieving http://maps.googleapis.com/maps/api/

geocode/json?sensor=false&address=Kokshetau+Inst ...

Retrieved 1749 characters { "results" : [

{u'status': u'OK', u'results': ... }

...

</pre>The first five locations are already in the database and so they

are skipped. The program scans to the point where it finds new

locations and starts retrieving them.The geoload.py program can be stopped at any time, and there is a counter

that you can use to limit the number of calls to the geocoding

API for each run. Given that the where.data only has a few hundred

data items, you should not run into the daily rate limit, but if you

had more data it might take several runs over several days to

get your database to have all of the geocoded data for your input.Once you have some data loaded into geodata.sqlite, you can

visualize the data using the geodump.py program. This

program reads the database and writes the file where.js

with the location, latitude, and longitude in the form of

executable JavaScript code. A run of the geodump.py program is as follows:<pre class="verbatim">Northeastern University, ... Boston, MA 02115, USA 42.3396998 -71.08975

Bradley University, 1501 ... Peoria, IL 61625, USA 40.6963857 -89.6160811

...

Technion, Viazman 87, Kesalsaba, 32000, Israel 32.7775 35.0216667

Monash University Clayton ... VIC 3800, Australia -37.9152113 145.134682

Kokshetau, Kazakhstan 53.2833333 69.3833333

...

12 records written to where.js

Open where.html to view the data in a browser

</pre>The file where.html consists of HTML and JavaScript to visualize

a Google map. It reads the most recent data in where.js to get

the data to be visualized. Here is the format of the where.js file:<pre class="verbatim">myData = [

[42.3396998,-71.08975, 'Northeastern Uni ... Boston, MA 02115'],

[40.6963857,-89.6160811, 'Bradley University, ... Peoria, IL 61625, USA'],

[32.7775,35.0216667, 'Technion, Viazman 87, Kesalsaba, 32000, Israel'],

...

];

</pre>This is a JavaScript variable that contains a list of lists.

The syntax for JavaScript list constants is very similar to

Python, so the syntax should be familiar to you.Simply open where.html in a browser to see the locations. You

can hover over each map pin to find the location that the

geocoding API returned for the user-entered input. If you

cannot see any data when you open the where.html file, you might

want to check the JavaScript or developer console for your browser.

<h2 class="section" id="sec188">15.2&#XA0;&#XA0;Visualizing networks and interconnections</h2>

<a id="hevea_default811"></a>In this application, we will perform some of the functions of a search

engine. We will first spider a small subset of the web and run

a simplified version of the Google page rank algorithm to

determine which pages are most highly connected, and then visualize

the page rank and connectivity of our small corner of the web.

We will use the D3 JavaScript visualization library

http://d3js.org/ to produce the visualization output.You can download and extract this application from:www.py4inf.com/code/pagerank.zip<div class="center"><img src="book022.png" /></div>The first program (spider.py) program crawls a web

site and pulls a series of pages into the

database (spider.sqlite), recording the links between pages.

You can restart the process at any time by removing the

spider.sqlite file and rerunning spider.py.<pre class="verbatim">Enter web url or enter: http://www.dr-chuck.com/

['http://www.dr-chuck.com']

How many pages:2

1 http://www.dr-chuck.com/ 12

2 http://www.dr-chuck.com/csev-blog/ 57

How many pages:

</pre>In this sample run, we told it to crawl a website and retrieve two

pages. If you restart the program and tell it to crawl more

pages, it will not re-crawl any pages already in the database. Upon

restart it goes to a random non-crawled page and starts there. So

each successive run of spider.py is additive.<pre class="verbatim">Enter web url or enter: http://www.dr-chuck.com/

['http://www.dr-chuck.com']

How many pages:3

3 http://www.dr-chuck.com/csev-blog 57

4 http://www.dr-chuck.com/dr-chuck/resume/speaking.htm 1

5 http://www.dr-chuck.com/dr-chuck/resume/index.htm 13

How many pages:

</pre>You can have multiple starting points in the same database&#X2014;within

the program, these are called &#X201C;webs&#X201D;. The spider

chooses randomly amongst all non-visited links across all

the webs as the next page to spider.If you want to dump the contents of the spider.sqlite file, you can

run spdump.py as follows:<pre class="verbatim">(5, None, 1.0, 3, u'http://www.dr-chuck.com/csev-blog')

(3, None, 1.0, 4, u'http://www.dr-chuck.com/dr-chuck/resume/speaking.htm')

(1, None, 1.0, 2, u'http://www.dr-chuck.com/csev-blog/')

(1, None, 1.0, 5, u'http://www.dr-chuck.com/dr-chuck/resume/index.htm')

4 rows.

</pre>This shows the number of incoming links, the old page rank, the new page

rank, the id of the page, and the url of the page. The spdump.py program

only shows pages that have at least one incoming link to them.Once you have a few pages in the database, you can run page rank on the

pages using the sprank.py program. You simply tell it how many page

rank iterations to run.<pre class="verbatim">How many iterations:2

1 0.546848992536

2 0.226714939664

[(1, 0.559), (2, 0.659), (3, 0.985), (4, 2.135), (5, 0.659)]

</pre>You can dump the database again to see that page rank has been updated:<pre class="verbatim">(5, 1.0, 0.985, 3, u'http://www.dr-chuck.com/csev-blog')

(3, 1.0, 2.135, 4, u'http://www.dr-chuck.com/dr-chuck/resume/speaking.htm')

(1, 1.0, 0.659, 2, u'http://www.dr-chuck.com/csev-blog/')

(1, 1.0, 0.659, 5, u'http://www.dr-chuck.com/dr-chuck/resume/index.htm')

4 rows.

</pre>You can run sprank.py as many times as you like and it will simply refine

the page rank each time you run it. You can even run sprank.py a few times

and then go spider a few more pages sith spider.py and then run sprank.py

to reconverge the page rank values. A search engine usually runs both the crawling and

ranking programs all the time.If you want to restart the page rank calculations without respidering the

web pages, you can use spreset.py and then restart sprank.py.<pre class="verbatim">How many iterations:50

1 0.546848992536

2 0.226714939664

3 0.0659516187242

4 0.0244199333

5 0.0102096489546

6 0.00610244329379

...

42 0.000109076928206

43 9.91987599002e-05

44 9.02151706798e-05

45 8.20451504471e-05

46 7.46150183837e-05

47 6.7857770908e-05

48 6.17124694224e-05

49 5.61236959327e-05

50 5.10410499467e-05

[(512, 0.0296), (1, 12.79), (2, 28.93), (3, 6.808), (4, 13.46)]

</pre>For each iteration of the page rank algorithm it prints the average

change in page rank per page. The network initially is quite

unbalanced and so the individual page rank values change wildly between

iterations. But in a few short iterations, the page rank converges. You

should run prank.py long enough that the page rank values converge.If you want to visualize the current top pages in terms of page rank,

run spjson.py to read the database and write the data for the

most highly linked pages in JSON format to be viewed in a

web browser.<pre class="verbatim">Creating JSON output on spider.json...

How many nodes? 30

Open force.html in a browser to view the visualization

</pre>You can view this data by opening the file force.html in your web browser.

This shows an automatic layout of the nodes and links. You can click and

drag any node and you can also double-click on a node to find the URL

that is represented by the node.If you rerun the other utilities, rerun spjson.py and

press refresh in the browser to get the new data from spider.json.

<h2 class="section" id="sec189">15.3&#XA0;&#XA0;Visualizing mail data</h2>

Up to this point in the book, you have become quite familiar with our

mbox-short.txt and mbox.txt data files. Now it is time to take

our analysis of email data to the next level. In the real world, sometimes you have to pull down mail data from servers.

That might take quite some time and the data might be inconsistent,

error-filled, and need a lot of cleanup or adjustment. In this section, we

work with an application that is the most complex so far and pull down nearly a

gigabyte of data and visualize it.<div class="center"><img src="book023.png" /></div>You can download this application from:www.py4inf.com/code/gmane.zipWe will be using data from a free email list archiving service called

www.gmane.org. This service is very popular with open source

projects because it provides a nice searchable archive of their

email activity. They also have a very liberal policy regarding accessing

their data through their API. They have no rate limits, but ask that you

don&#X2019;t overload their service and take only the data you need. You can read

gmane&#X2019;s terms and conditions at this page:http://gmane.org/export.phpIt is very important that you make use of the gmane.org data

responsibly by adding delays to your access of their services and spreading

long-running jobs over a longer period of time. Do not abuse this free service

and ruin it for the rest of us.When the Sakai email data was spidered using this software, it produced nearly

a Gigabyte of data and took a number of runs on several days.

The file README.txt in the above ZIP may have instructions as to how

you can download a pre-spidered copy of the content.sqlite file for

a majority of the Sakai email corpus so you don&#X2019;t have to spider for

five days just to run the programs. If you download the pre-spidered

content, you should still run the spidering process to catch up with

more recent messages.The first step is to spider the gmane repository. The base URL

is hard-coded in the gmane.py and is hard-coded to the Sakai

developer list. You can spider another repository by changing that

base url. Make sure to delete the content.sqlite file if you

switch the base url. The gmane.py file operates as a responsible caching spider in

that it runs slowly and retrieves one mail message per second so

as to avoid getting throttled by gmane. It stores all of

its data in a database and can be interrupted and restarted

as often as needed. It may take many hours to pull all the data

down. So you may need to restart several times.Here is a run of gmane.py retrieving the last five messages of the

Sakai developer list:<pre class="verbatim">How many messages:10

http://download.gmane.org/gmane.comp.cms.sakai.devel/51410/51411 9460

nealcaidin@sakaifoundation.org 2013-04-05 re: [building ...

http://download.gmane.org/gmane.comp.cms.sakai.devel/51411/51412 3379

samuelgutierrezjimenez@gmail.com 2013-04-06 re: [building ...

http://download.gmane.org/gmane.comp.cms.sakai.devel/51412/51413 9903

da1@vt.edu 2013-04-05 [building sakai] melete 2.9 oracle ...

http://download.gmane.org/gmane.comp.cms.sakai.devel/51413/51414 349265

m.shedid@elraed-it.com 2013-04-07 [building sakai] ...

http://download.gmane.org/gmane.comp.cms.sakai.devel/51414/51415 3481

samuelgutierrezjimenez@gmail.com 2013-04-07 re: ...

http://download.gmane.org/gmane.comp.cms.sakai.devel/51415/51416 0

Does not start with From

</pre>The program scans content.sqlite from one up to the first message number not

already spidered and starts spidering at that message. It continues spidering

until it has spidered the desired number of messages or it reaches a page

that does not appear to be a properly formatted message.Sometimes gmane.org is missing a message. Perhaps administrators can delete messages

or perhaps they get lost. If your spider stops, and it seems it has hit

a missing message, go into the SQLite Manager and add a row with the missing id leaving

all the other fields blank and restart gmane.py. This will unstick the

spidering process and allow it to continue. These empty messages will be ignored in the next

phase of the process.One nice thing is that once you have spidered all of the messages and have them in

content.sqlite, you can run gmane.py again to get new messages as

they are sent to the list. The content.sqlite data is pretty raw, with an inefficient data model,

and not compressed.

This is intentional as it allows you to look at content.sqlite

in the SQLite Manager to debug problems with the spidering process.

It would be a bad idea to run any queries against this database, as they

would be quite slow.The second process is to run the program gmodel.py. This program reads the raw

data from content.sqlite and produces a cleaned-up and well-modeled version of the

data in the file index.sqlite. This file will be much smaller (often 10X

smaller) than content.sqlite because it also compresses the header and body text.Each time gmodel.py runs it deletes and rebuilds index.sqlite, allowing

you to adjust its parameters and edit the mapping tables in content.sqlite to tweak the

data cleaning process. This is a sample run of gmodel.py. It prints a line out each time

250 mail messages are processed so you can see some progress happening, as this program may

run for a while processing nearly a Gigabyte of mail data.<pre class="verbatim">Loaded allsenders 1588 and mapping 28 dns mapping 1

1 2005-12-08T23:34:30-06:00 ggolden22@mac.com

251 2005-12-22T10:03:20-08:00 tpamsler@ucdavis.edu

501 2006-01-12T11:17:34-05:00 lance@indiana.edu

751 2006-01-24T11:13:28-08:00 vrajgopalan@ucmerced.edu

...

</pre>The gmodel.py program handles a number of data cleaning tasks.Domain names are truncated to two levels for .com, .org, .edu, and .net.

Other domain names are truncated to three levels. So si.umich.edu becomes

umich.edu and caret.cam.ac.uk becomes cam.ac.uk. Email addresses are also

forced to lower case, and some of the @gmane.org address like the following<pre class="verbatim"> arwhyte-63aXycvo3TyHXe+LvDLADg@public.gmane.org

</pre>are converted to the real address whenever there is a matching real email

address elsewhere in the message corpus.In the content.sqlite database there are two tables that allow

you to map both domain names and individual email addresses that change over

the lifetime of the email list. For example, Steve Githens used the following

email addresses as he changed jobs over the life of the Sakai developer list:<pre class="verbatim">s-githens@northwestern.edu

sgithens@cam.ac.uk

swgithen@mtu.edu

</pre>We can add two entries to the Mapping table in content.sqlite so

gmodel.py will map all three to one address:<pre class="verbatim">s-githens@northwestern.edu -> swgithen@mtu.edu

sgithens@cam.ac.uk -> swgithen@mtu.edu

</pre>You can also make similar entries in the DNSMapping table if there are multiple

DNS names you want mapped to a single DNS. The following mapping was added to the Sakai data:<pre class="verbatim">iupui.edu -> indiana.edu

</pre>so all the accounts from the various Indiana University campuses are tracked together.You can rerun the gmodel.py over and over as you look at the data, and add mappings

to make the data cleaner and cleaner. When you are done, you will have a nicely

indexed version of the email in index.sqlite. This is the file to use to do data

analysis. With this file, data analysis will be really quick.The first, simplest data analysis is to determine "who sent the most mail?" and "which

organization sent the most mail"? This is done using gbasic.py:<pre class="verbatim">How many to dump? 5

Loaded messages= 51330 subjects= 25033 senders= 1584

Top 5 Email list participants

steve.swinsburg@gmail.com 2657

azeckoski@unicon.net 1742

ieb@tfd.co.uk 1591

csev@umich.edu 1304

david.horwitz@uct.ac.za 1184

Top 5 Email list organizations

gmail.com 7339

umich.edu 6243

uct.ac.za 2451

indiana.edu 2258

unicon.net 2055

</pre>Note how much more quickly gbasic.py runs compared to gmane.py

or even gmodel.py. They are all working on the same data, but

gbasic.py is using the compressed and normalized data in

index.sqlite. If you have a lot of data to manage, a multistep

process like the one in this application may take a little longer to develop,

but will save you a lot of time when you really start to explore

and visualize your data.You can produce a simple visualization of the word frequency in the subject lines

in the file gword.py:<pre class="verbatim">Range of counts: 33229 129

Output written to gword.js

</pre>This produces the file gword.js which you can visualize using

gword.htm to produce a word cloud similar to the one at the beginning

of this section.A second visualization is produced by gline.py. It computes email

participation by organizations over time.<pre class="verbatim">Loaded messages= 51330 subjects= 25033 senders= 1584

Top 10 Oranizations

['gmail.com', 'umich.edu', 'uct.ac.za', 'indiana.edu',

'unicon.net', 'tfd.co.uk', 'berkeley.edu', 'longsight.com',

'stanford.edu', 'ox.ac.uk']

Output written to gline.js

</pre>Its output is written to gline.js which is visualized using gline.htm.<div class="center"><img src="book024.png" /></div>This is a relatively complex and sophisticated application and

has features to do some real data retrieval, cleaning, and visualization.

<hr />

</body>

</html>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FilesExpand file tree

book016.html

Latest commit

History

book016.html

File metadata and controls