java - Hbase Scheme design- Best Practice -

-

i have switched hbase rdbms handling millions of records.. newbie not sure efficient way of designing hbase scheme. actually, scenario have text files have hundred, thousands , millions records have read , store hbase. so, there 2 set of text files(rawdata file, label file) linked each other belong same user, these files have made 2 separate tables(rawdata , label) , storing info there. rawdata file , rawdata table this:

enter image description here enter image description here

so can see in rawdata table have row key file name of text file( 01-01-all-data.txt) row number of each row of textfile. , column family random 'r' , column qualifiers columns of text files , value values of column. how inserting record in table , have third table(mapfile) store name of textfile row key user id of user column qualifier , total number of records of textfile value looks this:

            01-01-all-data.txt       column=m:1, timestamp=1375189274467, value=146209

i use mapfile table in order read rawdata table row row..

what suggestion kind hbase schema? proper way? or doesn't make sense in hbase concepts?

furthermore, worths mention taking around 3 mins in inserting 21 mbs file 146207 rows in hbase.

please advice.

thanks

although don't find wrong current schema, it's appropriate or not can decided after analyzing use case , frequent access pattern. correct not appropriate, imho. since don't have idea suggestions may sound incorrect. please let me know if case. i'll update answer accordingly. here go,

does make sense(keeping data , access pattern in mind) have 1 table 3 column families :

  • rd - rawdata file have columns of file
  • lf - label file columns of file, and
  • mf - mapfile having 1 column holding number of records of textfile.

use userid rowkey. unique , doesn't lengthy. design bypass overhead of shunting 1 table while fetching data.

few more suggestions :

  • if userids monotonically increasing hash rowkeys don't suffer regionserver hotspotting.
  • you create pre-splitted tables in order better distribution.
  • shorten column names if possible.
  • keep number of version low possible.

furthermore, worths mention taking around 3 mins in inserting 21 mbs file 146207 rows in hbase.

how inserting data?mapreduce or normal java+hbase api?what cluster size?configuration , specs?

you might find these links useful :

hth


Comments

Post a Comment

Popular posts from this blog

c++ - Creating new partition disk winapi -

-
i'am trying create new partition , mount volume new, thought createfile let me this, code : lpctstr lpfilename=l"\\\\.\\device\\harddisk0\\partition3"; handle handl=createfile( lpfilename, generic_read | generic_write, file_share_read | file_share_write, null, create_always, file_attribute_normal, null ); if (handl==invalid_handle_value) { qdebug()<<"handl invalid"<<" error"<<getlasterror();} bool success = definedosdevice(ddd_raw_target_path,l"i:",l"\\device\\harddisk0\\partition3"); if(!success) qdebug()<<" definedosdevice failed "<<getlasterror(); bflag = getvolumenameforvolumemountpoint( l"i:\\", // input volume mount point or directory /** u in dir
Read more

Android Prevent Bluetooth Pairing Dialog -

-
i'm developing internal application uses bluetooth printing. want bluetooth pairing occur without user input. have managed working trapping android.bluetooth.device.action.pairing_request broadcast. in broadcast receiver call setpin method, , pairing works ok, bluetoothpairingdialog displayed second or two, disappears - see link below. https://github.com/android/platform_packages_apps_settings/blob/master/src/com/android/settings/bluetooth/bluetoothpairingdialog.java since broadcast non-ordered, can't call abortbroadcast() , , wondering if there other way prevent pairing dialog appearing. can hook window manager in way? i haven't been able come way without modifying sdk. if you're oem, it's easy (i'm on 4.3): in packages/apps/settings/androidmanifest.xml, comment intent filter pairing dialog: <activity android:name=".bluetooth.bluetoothpairingdialog" android:label="@string/bluetooth_pairing_request"
Read more

VBA function to include CDATA -

-
i have following function include cdata: function cdatasection() dim objdom domdocument dim objkmlrootelement ikmldomelement dim objkmlelement ikmldomelement dim cdata ikmldomcdatasection set objdom = new domdocument set objkmlrootelement = objdom.createelement("balloonstyle") objdom.appendchild objkmlrootelement set objkmlelement = objdom.createelement("text") objkmlrootelement.appendchild objkmlelement set cdata = objdom.createcdatasection("text") cdata.data = "<![cdata[<b>latitude = $[latitude]</b>?]]>;" end function when run above, getting error "user defined data type not found" function. yes ripster says find library reference. vba window tools>references for list of relevant references see... http://msdn.microsoft.com/en-us/library/windows/desktop/ms763701%28v=vs.85%29.aspx however, if have hard time tracking down reference, replace dim sta
Read more