WebFaction
Community site: login faq

I am trying to follow the instructions here (https://groups.google.com/forum/?fromgroups=#!msg/geodjango/butW7zJQhSw/FuRQ09VbI3cJ) to add US zipcode information to my database so that I can query using distance. However, the operation suggested here to add the data is too memory intensive to be run without crashing on my Webfaction server.

Can anyone help me figure out how to get this information into my database so that I can make distance queries? I have been working on this for a while and have not arrived at a solution, even with the help of the Webfaction support team. Thanks for any ideas!


EDIT: adding commands that I need to run

In my python interpreter:

>>> from django.contrib.gis.utils import LayerMapping
>>> from location.models import Zipcode
>>> lm = LayerMapping(Zipcode, '/home/myaccount/webapps/myapp_static/tl_2008_us_zcta5.shp',
... {'code':'ZCTA5CE',
...  'mpoly':'MULTIPOLYGON',
... })
>>> lm.save()

This is the 'Zipcode' model mentioned above:

class Zipcode(models.Model):
    code = models.CharField(max_length=5, db_index=True)
    mpoly = models.MultiPolygonField()
    objects = models.GeoManager()

    def __unicode__(self):
        return self.code

Thanks for any ideas!

asked 17 Jun '13, 17:04

Nick_B
111014
accept rate: 0%

edited 21 Jun '13, 14:42


When using LayerMapping, make sure you have DEBUG set to False, otherwise it will use tons of memory. Have you tried loading the shapefiles into the database with shp2pgsql? I'm not sure if this will work, since you are working with files of more than 600MB, but it's worth a shot. (shp2pgsql is installed by default on Webfaction)

Here's some info on it: http://suite.opengeo.org/docs/dataadmin/pgGettingStarted/shp2pgsql.html

permanent link

answered 22 Jun '13, 02:20

djaplat_sftp
262
accept rate: 25%

Thanks that is immensely helpful. I did not realize how much a difference the DEBUG settings would have. It worked

(27 Jun '13, 20:17) Nick_B

If processing the data on your WebFaction server causes you to exceed your memory limit, then you could instead process it locally, on your own computer, and then import the processed data directly into your database.

permanent link

answered 17 Jun '13, 17:20

seanf
12.2k41836
accept rate: 37%

Are there any other options for me? I am having a lot of difficulty getting PostGIS installed locally. I would gladly pay extra to have a tech help me get this information into the database rather than spend days trying to get this set up locally...

(21 Jun '13, 13:07) Nick_B

Our support team would not be able to circumvent the RAM limitations - there's only so much RAM available on our servers.

Perhaps if you can share the code that you are using to import your data, we might be able to see a way to make it less memory-intensive. For example, if you're doing a single huge DB insert, perhaps you could break that into smaller batches?

(21 Jun '13, 13:21) seanf

Hi @seanf thanks for your comment. I added the operation that I need to perform above. I would be very thankful for any ideas that could help! Thanks!

(21 Jun '13, 14:41) Nick_B

Ok, so it looks like you're attempting to import a single huge file. Splitting into smaller pieces (which you could then import separately) would be the logical solution. There's some info here that might help: Split or save a subset of a ESRI Shape SHP file to a new file?

(21 Jun '13, 15:18) seanf
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×909
×337
×1

question asked: 17 Jun '13, 17:04

question was seen: 2,703 times

last updated: 27 Jun '13, 20:17

WEBFACTION
REACH US
SUPPORT
AFFILIATE PROGRAM
LEGAL
© COPYRIGHT 2003-2019 SWARMA LIMITED - WEBFACTION IS A SERVICE OF SWARMA LIMITED
REGISTERED IN ENGLAND AND WALES 5729350 - VAT REGISTRATION NUMBER 877397162
5TH FLOOR, THE OLD VINYL FACTORY, HAYES, UB3 1HA, UNITED KINGDOM