#
RHIPE(phonetic spelling: hree-pay’ [1]) is a java package that integrates the R environment with Hadoop, the open source implementation of Google’s mapreduce. Using RHIPE it is possible to code map-reduce algorithms in R e.g [1] This is greek for a moment in time. See here for pronunciation: Greek Lexicon
m <- expression({
y <- unlist(strsplit(unlist(map.values),"[[:space:]]+"))
sapply(y,function(r) rhcollect(r,T))
## instead of the previous line, you could also do
## z <- table(sapply(y,function(r) rhcollect(r,T)))
## sapply(names(z), function(r) rhcollect(r, z[r]))
})
r <- expression(
pre={
count=0
},
reduce={
count <- sum(as.numeric(unlist(reduce.values)),count)
},post={
rhcollect(reduce.key,count)
})
z=rhmr(map=m,reduce=r,comb=T,inout=c("text","sequence"),ifolder="/tmp/50mil",ofolder='/tmp/tof')
rhex(z)
Or just, load Rhipe and type
rhwordcount(infolder,outfolder)
where infolder is the input file(or folder of files) of words(text file) and outfolder is
the destination directory.
Mailing list is hosted on Google Groups. The url is http://groups.google.com/group/rhipe . Your first post will be moderated.
Well, a useful subset, but not all, see Data Types
For more information about what RHIPE is and not, read the FAQ. Please note, this does not work on Mac OS X Snow Leopard.
Source
The source code is present on Git, go here http://github.com/saptarshiguha/RHIPE/
To check out the current version, install git
git clone git://github.com/saptarshiguha/RHIPE.gitTo download version X e.g 0.45
git clone git://github.com/saptarshiguha/RHIPE.git
git checkout 0.45The current version is always the master.
Versions
Read the documentation for installation. Current is the latest version.
| Version | Download | |
|---|---|---|
| 0.61 | Rhipe_0.61.tar.gz | Wed Sep 08 03:12:20 EDT 2010 |
| 0.60 | Rhipe_0.60.tar.gz | Fri May 28 17:13:16 EDT 2010 |
| 0.59 | Rhipe_0.59.tar.gz | Tue May 04 18:15:26 EDT 2010 |
| 0.58 | Rhipe_0.58.tar.gz | Fri Apr 09 01:41:48 EDT 2010 |
| 0.57 | Rhipe_0.57.tar.gz | Sat Mar 20 11:20:21 EDT 2010 |
| 0.56 | Rhipe_0.56.tar.gz | Sat Feb 27 17:04:44 EST 2010 |
| 0.55 | Rhipe\_0.55.tar.gz | Sun Feb 21 00:49:04 EST 2010 |
| 0.54 | Rhipe\_0.54.tar.gz | |
| 0.53 | Rhipe\_0.53.tar.gz | |
| 0.52 | Rhipe\_0.52.tar.gz | |
| 0.51 | Rhipe\_0.51.tar.gz | |
| 0.5 | Rhipe\_0.5.tar.gz | |
| 0.44 | rhipe.0.44.tgz |
See the documentation.
The documentation can be found here. PDF version can be found here
sguha -AT- purdue -DOT- edu
- Version 0.61, minor=3
- Some modifications made to sorting of keys. Works now.
- Manual completely re-written
- The experimental java as a server episode has been re-written.
- Version 0.61, minor=2
- Ordering of numeric and alphabetical keys (not default)
- Version 0.61
- Added a partitioner that partitions on the i’th element of a scalar vector (strings, numerics and integers)
- Now version 0.60
- Added asynch options to
rhex, so jobs can run in the background freeing the R console. The return value can be used to monitor job progress. See Miscellaneous Commands for more information.
- Added
rhcpandrhmvto copy and moves files when both source and destination are on the HDFS (thanks to Jeff Li)
- Some bugs in the comparator - fixed.
- fixed comparators,
rhgetkeyworking again. (0.59-2)
- Fixed a bug in
rhlapply, would not read in data. Thanks to eddyu rhoptions()$versionnow has displays major, minor , date and notes. I added this since i make changes to RHIPE but keep the version the same.
- Less memory allocation in the key/value(s).
rhreadnow does not do a mapreduce job to convert sequence files to binary. Also has aheadlike function.- if multicore is installed, then running rhread(..,mc=TRUE) will deserialize in parallel, which might or might be slower …
rheztakes an optionmapredwhich is of the same form asmapredinrhmr. This will override themapredvalue inrhmr.rhgetkeytakes a parameterskipto read in large databases, also no need for trailing “*”.
- moved to protobuf-2.3
rhlscan now recurserhreadnow need only take a folder (no need forrhmap.sqsto read map files). Use thetypeargument to specify sequence(or text) files or map files.- Similarly
rhmrdoes not needrhmap.sqs
rhreadtakes amaxargument that reads in only max number of key-value pairsrhexpasses all extra arguments to the system command.
- Combiner bug fixed, it’s still needs to be tested. Mail if numbers do not match.
- Fixed combiner, still alpha, but it halves the wordcount speed. The combiner logic is run in the R interpreter C code. However it is still alpha, so if you get erroneous results kindly report them back.
- Also fixed a buffer overflow in
main.c. Thanks to Will Nolan. - Values and Keys can be now be upto 256MB.
- EC2 now works!
- Counters are returned to the R session (for
rhmronly). That is the return value ofrhmris a list, the first element indicates success/failure and the second are all the counters visible in the job UI.
- Fixed a bug where errors in R code were not appearing. Somewhat fixed. Version stays the same.
Released version 0.54
- Introduce a Hadoop Map File Outputformat and functions for reading a key from map files(see help on rhmr and misc functions)
- Fixed a bug for the case when no reducer is specified but RHIPE java code threw an exception.
- Release **Version 0.53**
- Bug fixes:
- Inserted R\_CStackLimits, since I’m using Protobuf a threaded library, it was upsetting R.
- Removed Rf\_duplicate
- Data types have been enhanced, now allows scalar vectors with attributes. Experimental.
- A result of which can now write data.frames and read them back in.
- Impose 64MB key,value serialization limit(workaround to come in future). Objects bigger than this will be written successfully,but will fail to read and will cause the job to fail.
rhcounter,available in mapreduce code, is more versatile. Previously, ‘,’ in the counter names would upset Hadoop. Not anymore, see documentation forrhmr
- Failed when running RHIPE from different UID’s. Now writes to
/tmp/logger-UID. Version number is still the same
- Removed the dependency on rJava. Getting it to work with Hadoop classpaths caused to much grief. The actualy RHIPE program remains unchanged but the client handler (R package) is a bit slower(?)
- Names are only read for VECSXP (list objects), because of a strange bug.
- Moved to Hadoop 0.20
- Uses protobuf for serialization, fewer R types allowed
- Does not depend on Rserve, single R package to install
- Web site revamped. Beginning with the current version, the entire manual is in PDF or can be accessed at the documentation link.
- Source code is available on Git, go to the download page for instructions.
- Stopped seeding via secure random generator, so the user will have
to seed it to avoid correlated streams. On RHEL linux
when running
rhlapplyon 145K+ tasks,/dev/randomwould block.