To initialize one of my applications, I need to read over a large (~78MB) XML file and populate a database table. This XML file has about 300,000 entries of interest. I've put the initialization code in my application's
While executing, this code peaked at a whopping 242MB of memory, more than three times the size of the input file! [Edit: and the process persisted until I manually killed it.]
Thankfully, I'll never need to run this function again. But I might need to run functions like it, especially if I'm parsing more XML in the future. Quite simply, my question is: why is this happening? This is what I've considered so far:
Otherwise, I don't know what could be causing this. I'd really appreciate some help. Thanks for reading! asked 18 Oct '11, 03:14 phokai |
It's hard to say why the process is taking so much memory. There could be a memory leak somewhere in the code. I'd recommend reading our docs on reducing memory consumption, if you haven't already done so. Our system will kill a process that's consuming too much memory if it has been running more than 5 minutes. So, if you only need to run this once, and it takes less than 5 minutes, it should be ok. answered 18 Oct '11, 03:59 todork Call me naive, but I didn't realize that Python could have memory leaks. What sorts of code could cause memory leaks?
(18 Oct '11, 10:29)
phokai
1
That's a broad question, so here is a broad answer: Python memory leaks :)
(18 Oct '11, 12:57)
seanf
I had already tried that exact query and found three possible causes, according to the first result: """ 1. some low level C library is leaking 2. your Python code have global lists or dicts that grow over time, and you forgot to remove the objects after use 3. there are some reference cycles in your app """ I was hoping for some advice more specific to Django development, but perhaps there isn't any.
(18 Oct '11, 19:43)
phokai
|