Prev: about Telnetlib problem
Next: Anything like "Effective Java" for Python?
From: Richard on 10 Mar 2010 09:29
I want to atomically write to a file so have been using temporary
files and renaming:

temp = tempfile.NamedTemporaryFile(delete=False)
temp.file.write(data)
temp.file.close()
os.rename(temp.name, output_file)

This worked but after 39567 files os.rename raises an OSError: [Errno
31] Too many links

I can still create files in this directory so I doubt it is a platform
limitation.
Can you tell what is wrong? Am I not freeing the temporary file
resources properly?

thanks,
Richard


From: Thomas Guettler on 10 Mar 2010 11:08
Richard wrote:
> I want to atomically write to a file so have been using temporary
> files and renaming:
>
> temp = tempfile.NamedTemporaryFile(delete=False)
> temp.file.write(data)
> temp.file.close()
> os.rename(temp.name, output_file)
>
> This worked but after 39567 files os.rename raises an OSError: [Errno
> 31] Too many links
>
> I can still create files in this directory so I doubt it is a platform
> limitation.
> Can you tell what is wrong? Am I not freeing the temporary file
> resources properly?

If you are on linux you can have a look at the open file descriptors of
a running process like this:

ls -l /proc/PID/fd/

But I guess it is a limitation of your filesystem. What do you use?

I once had this problem with ext2. It has a low limit for
subdirectories.

With xfs the limits are much greater.

Thomas


--
Thomas Guettler, http://www.thomas-guettler.de/
E-Mail: guettli (*) thomas-guettler + de
From: Richard on 11 Mar 2010 01:22
On Mar 11, 3:08 am, Thomas Guettler <h...(a)tbz-pariv.de> wrote:
> Richard wrote:
> > I want to atomically write to a file so have been using temporary
> > files and renaming:
>
> > temp = tempfile.NamedTemporaryFile(delete=False)
> > temp.file.write(data)
> > temp.file.close()
> > os.rename(temp.name, output_file)
>
> > This worked but after 39567 files os.rename raises an OSError: [Errno
> > 31] Too many links
>
> > I can still create files in this directory so I doubt it is a platform
> > limitation.
> > Can you tell what is wrong? Am I not freeing the temporary file
> > resources properly?
>
> If you are on linux you can have a look at the open file descriptors of
> a running process like this:
>
> ls -l /proc/PID/fd/
>
> But I guess it is a limitation of your filesystem. What do you use?
>
> I once had this problem with ext2. It has a low limit for
> subdirectories.
>
> With xfs the limits are much greater.
>
>   Thomas
>
> --
> Thomas Guettler,http://www.thomas-guettler.de/
> E-Mail: guettli (*) thomas-guettler + de


I am using Ubuntu 9.10 with ext3, which I believe has a limit for the
number of subdirectories but not files.

Thanks for the open file descriptor tip - I will check that out.

Richard
 | 
Pages: 1
Prev: about Telnetlib problem
Next: Anything like "Effective Java" for Python?