Thread scheduling on an SMP
Hello:
I have a machine with 4 processors, and I wold like to make the
following: Set 1 processor to do general stuff using the default scheduler and the other three working on my program exclusively, with the realtime or my own thread scheduler.
Is this possible?, thanks for any comment or links to documentation
you can give me.
Darío
-
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
file deletion
Does anybody know of any useful code or c commands that
you can use to search for duplicate files within a linux/unix directory and its subdirectories and remove them?
Thanks
Andrew
Re: file deletion
On Fri, 2004-12-24 13:14:11 -0000, Andy <[email protected]> wrote in message <003f01c4e9ba$7679d020$316c4ed5@j0s6l8>:
Quoted 3 lines
Does anybody know of any useful code or c commands that
you can use to search for duplicate files within a linux/unix directory and its subdirectories and remove them?
As long as you believe in cryptographical hashes, something like this should do the trick:
SOME_DIR=/some/directory
find "${SOME_DIR}" -type f -exec sha1sum {} \; | \
sort | \
uniq -c | \
sed -e 's/^[[:space:]]*//g' | \
egrep -v '\<1\>' | \
cut -f 4- -d ' ' | \
while read DOUBLE_FILE_NAME; do
rm -f "${SOME_DIR}/${DOUBLE_FILE_NAME}"
doneThat's untested, but should work (it's essentially a one-liner). However, it removes *all* instances of files which are believed to be identical...
MfG, JBG
Signature · 4 lines
Jan-Benedict Glaw [email protected] . +49-172-7608481 _ O _ "Eine Freie Meinung in einem Freien Kopf | Gegen Zensur | Gegen Krieg _ _ O fuer einen Freien Staat voll Freier Bürger" | im Internet! | im Irak! O O O ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));
Re: file deletion
In reply to AndyQuoted 5 lines · Andy
Does anybody know of any useful code or c commands that
you can use to search for duplicate files within a linux/unix directory and its subdirectories and remove them?
Thanks
Andrew
Ehm.. Personally no.. However there is ftw, opendir, readdir, stat, fstat... etc.. Ones you are able to access all the directory entries you have to make a decision how you want to compare e.g. md5, crc32, only size or name etc... Then there is the issue of choosing a optimal ADT and access/retrieval algo.
If you dont have to c code the program but just looking for a solution I would most certainly go for a:find -type f -exec md5sum '{}' \; >> md5.log
and parse the md5.log with a simple shell or awk script. That would save many headache's.. Plus you don't have to reinvent find...
J.
Re: file deletion
In reply to AndyQuoted 3 lines · Andy
Does anybody know of any useful code or c commands that
you can use to search for duplicate files within a linux/unix directory and its subdirectories and remove them?
First, you have to define "duplicate". Also, once you've decided that two or more files are duplicates, you have to decide which one you wish to keep and which ones are to be deleted.
If you consider any files with identical MD5 hashes as duplicates, and don't care about which one is kept, you could use something like:
find . -type f -exec md5sum {} \; | sort | uniq -w32 -d | cut -b 35- | \
while read file ; do rm -- "$file" ; done
However, if you have 3 or more copies of a given file, the above will only delete one of them ("uniq -d ..." only prints one instance of each duplicate line; "uniq -D ..." prints all instances, which would result in all copies being removed; there isn't an all-but-one option).
Signature · 1 line
Glynn Clements <[email protected]>
Re: file deletion
Hello Glynn et al,
Quoted 14 lines · On Thu, 30 Dec 2004 11:26:21 +0000 Glynn Clements <[email protected]>…
Quoted 3 lines · Andy
Does anybody know of any useful code or c commands that
you can use to search for duplicate files within a linux/unix directory and its subdirectories and remove them?First, you have to define "duplicate". Also, once you've decided that two or more files are duplicates, you have to decide which one you wish to keep and which ones are to be deleted.
If you consider any files with identical MD5 hashes as duplicates, and don't care about which one is kept, you could use something like:
Quoted 7 lines
find . -type f -exec md5sum {} \; | sort | uniq -w32 -d | cut -b 35- | \
while read file ; do rm -- "$file" ; doneHowever, if you have 3 or more copies of a given file, the above will only delete one of them ("uniq -d ..." only prints one instance of each duplicate line; "uniq -D ..." prints all instances, which would result in all copies being removed; there isn't an all-but-one option).
An idea would also be to remove all-but-one and create hardlinks in place of the removed ones, maybe? Of course it would depend Andy's answers to the questions raised by Glynn: what means 'duplicate' and what to do w/ those dups (even why are you checkinf for dups and what are you expecting to be able to do?).
Regards,
Signature · 1 line
wwp
Re: file deletion
Gents,
thanks for this, this script seems to do the trick. Actually what I meant by "duplicate" but didn't clarify was identical name and file size, dates do not matter. What does md5sum {} do?
Thanks
Andy
----- Original Message -----
From: "wwp" <[email protected]>
To: <[email protected]>
Sent: Thursday, December 30, 2004 12:04 PM
Subject: Re: file deletion
Quoted 4 lines
Hello Glynn et al,
On Thu, 30 Dec 2004 11:26:21 +0000 Glynn Clements
Quoted 6 lines · <[email protected]> wrote:
Quoted 6 lines
Quoted 2 lines · Andy
Does anybody know of any useful code or c commands that you can use to search for duplicate files within a linux/unix
directory and
Quoted 12 lines
Quoted 12 lines
Quoted 1 line
its subdirectories and remove them?
First, you have to define "duplicate". Also, once you've decided that two or more files are duplicates, you have to decide which one you wish to keep and which ones are to be deleted.
If you consider any files with identical MD5 hashes as duplicates, and don't care about which one is kept, you could use something like:
find . -type f -exec md5sum {} \; | sort | uniq -w32 -d | cut -b 35- | \
while read file ; do rm -- "$file" ; done
Quoted 6 lines
Quoted 4 lines
However, if you have 3 or more copies of a given file, the above will only delete one of them ("uniq -d ..." only prints one instance of each duplicate line; "uniq -D ..." prints all instances, which would result in all copies being removed; there isn't an all-but-one option).
An idea would also be to remove all-but-one and create hardlinks in place
of
Quoted 6 lines
the removed ones, maybe? Of course it would depend Andy's answers to the questions raised by Glynn: what means 'duplicate' and what to do w/ those dups (even why are you checkinf for dups and what are you expecting to be able to do?).
Quoted 6 lines
Regards,
--
wwp
-
To unsubscribe from this list: send the line "unsubscribe
linux-c-programming" in
Quoted 3 lines
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: file deletion
On Sat, 2005-01-01 11:09:37 -0000, Andy <[email protected]> wrote in message <000b01c4eff2$623ff7e0$8a7e4ed5@j0s6l8>:
Quoted 5 lines
Gents,
thanks for this, this script seems to do the trick. Actually what I meant by "duplicate" but didn't clarify was identical name and file size, dates do not matter. What does md5sum {} do?
The "find" command will substitute "{}" inside an "-exec ... ;" expression with the filename it currently checks. So this is only a md5sum call with a given filename.
MfG, JBG
Signature · 4 lines
Jan-Benedict Glaw [email protected] . +49-172-7608481 _ O _ "Eine Freie Meinung in einem Freien Kopf | Gegen Zensur | Gegen Krieg _ _ O fuer einen Freien Staat voll Freier Bürger" | im Internet! | im Irak! O O O ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));
Linux source code: malloc.c
In reply to AndyHi,
Where can I see the implementation of free() and malloc() in Linux source code? I am not able to locate the file malloc.c.
thanks,
venkatesh
Re: Linux source code: malloc.c
Quoted 2 lines
Where can I see the implementation of
free()andmalloc()in Linux source code? I am not able to locate the filemalloc.c.
That's because malloc() isn't in Linux-the-kernel: it's in glibc. The file you're looking for is malloc/malloc.c in the glibc tarball.
Steven Smith.
Signature · 2 lines
One day, I'm going to get an Alice-bot to answer all my email for me, and see how long it takes people to notice.
Re: Linux source code: malloc.c
Mensaje citado por: Steven Smith <[email protected]>:
Quoted 5 lines
Quoted 2 lines
Where can I see the implementation of
free()andmalloc()in Linux source code? I am not able to locate the filemalloc.c.That's because
malloc()isn't in Linux-the-kernel: it's in glibc. The file you're looking for ismalloc/malloc.cin the glibc tarball.
Except the kernel implements its own "kmalloc", which if I'm not wrong is called upon by libc's malloc
-------------------------------------------------
www.correo.unam.mx
UNAMonos Comunicándonos
-
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Linux source code: malloc.c
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Where can I see the implementation of
free()andmalloc()in Linux source code? I am not able to locate the filemalloc.c.That's because
malloc()isn't in Linux-the-kernel: it's in glibc. The file you're looking for ismalloc/malloc.cin the glibc tarball.Except the kernel implements its own "kmalloc", which if I'm not wrong is called upon by libc's malloc
Um, sort of... the kernel does implement a function kmalloc, but it's not the same as the user-exposed one. For a start, memory allocated by kmalloc is only accessible from kernel space, rather than user programs.
The kernel does provide two main primitives for userspace programs to allocated memory: mmap and brk. These are both fairly limited, though. mmap can only cope with multiples of the page size (4096 bytes on x86), while brk enforces a stack-like discipline.
sys_mmap and sys_brk do call upon kmalloc themselves, though.
Steven.
If I am reading your question correctly, you are talking about schedular affinity. You can set which processors you would like to assign pid's to. You need a kernel after 2.5.8 for thiese functions...
Excerpt:
#include <sched.h>
int sched_setaffinity(pid_t pid, unsigned int len, unsigned long *mask);
int sched_getaffinity(pid_t pid, unsigned int len, unsigned long *mask);
DESCRIPTIONsched_setaffinity sets the CPU affinity mask of the process denoted by pid. If pid is zero, then the current process is used.
The affinity mask is represented by the bitmask stored in mask. The least significant bit corresponds to the first logical processor number on the system, while the most significant bit corresponds to the last logical processor number on the system. A set bit corresponds to a legally schedulable CPU while an unset bit corresponds to an illegally schedulable CPU. In other words, a process is bound to and will only run on processors whose corresponding bit is set. Usually, all bits in the mask are set.
The argument len is the length of the data pointed to by mask. Normally this is the size of a word on the system. For compatibility with future versions of the Linux kernel, since this size can change, the bitmask supplied must be at least as large as the affinity mask stored in the kernel.
The function sched_getaffinity writes into the pointer supplied by mask that is size len the affinity mask of process pid. If pid is zero, then the mask of the current process is returned.
Quoted 14 lines · Darío Mariani
Hello:
I have a machine with 4 processors, and I wold like to make the
following: Set 1 processor to do general stuff using the default scheduler and the other three working on my program exclusively, with the realtime or my own thread scheduler.
Is this possible?, thanks for any comment or links to documentation
you can give me.Darío
-
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 8 lines
| Richard Nairn Specializing in Linux | Nairn Consulting Web / Database Solutions | Calgary, AB | [email protected] - To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html