TinyMUX: Info Storage
-
Re: Information Storage Question
So this came up in my own code. Some info for TinyMUX users with standard installs:
-
Data in a single attribute: 8000 characters counting the attribute name, plus attribute settings and seemingly some other weirdness, so really somewhere in the neighborhood of 7900 characters if you don't want to risk cutting it off.
-
Max attribute name length without data loss: 63
-
Attributes on a single object: I got up to 2665 in a number of tests with full-named attributes all filled with 7900+ characters of data, but some of my other tests I managed only as high as 2513, so decided to cut it off at 2500. Again, some kind of attribute naming scheme weirdness.
-
Number of attributes able to be stored on an object does not appear to have anything to do with the object's memory as reported by objmem() - you get the same results if you just put in a 1 for the value and use simple numeric attribute names.
Testing methodology:
@wait me=th attrcnt(#672) @dolist/notify lnum(700)=@set/quiet #672=_L.##:1Repeat in increments of 700 until object won't take anymore. MUX fails silently so you'll have to keep an eye on it.
Next test:
@dolist/notify lnum(700)=@set/quiet #672=_L.##.[iter(lnum(rand(sub(59,strlen(##)))),pickrand(a b c d e f g h i j k l m n o p q r s t u v w x y z),,@@)]:[repeat(0123456789, 799)]Screwing around like that got me the "magic" numbers above.
I decided I didn't want to worry about this crap so I'd write code to do it for me. That code had the following requirements:
- Pure MUX. I leave the SQL to someone willing to write something that will do it right in a non-blocking fashion.
- Store at least 20k attributes.
- Do it all behind-the-scenes, no wizard intervention required when things get over-stuffed.
- Must not lose data. MUX's "I eat data for breakfast" habit bugged me.
- Must function transparently to the coder using it - so just like set().
- Be expandable in case 20k isn't enough.
From that came this code, including all the documentation I could think of:
@create Data Storage Functions <DSF>=10 @set DSF=safe inherit @@ Unique key goes here. Required if you want to call this from code without @@ opening the function up to everyone. &vK DSF=UNIQUE KEY HERE @@ Exercise caution expanding this. On MUX with extensive testing I've found that @@ you can push this up to 2662, but my tests were inconclusive when I started @@ playing with attribute length. Data loss is possible with values higher than 2550. @@ Playing it safe and setting this low enough that there's lots of room. &d.max-attributes DSF=2500 @@ From my tests, these work. 7900 because I ran into trouble with numbers past @@ 7940 with long attribute names. Rounding a bit because I like round numbers. &d.max-attr-name-length DSF=63 &d.max-attr-value-length DSF=7900 @@ Who can run this? Strong recommendation: at least wizard. Should take care of @@ your inherit objects and code stuff - we only want this being called by @@ system code if possible. Since I couldn't find a lock method that actually @@ worked I stuck in a unique key. Fix this if you know what the heck you're @@ doing. @@ @@ Arg: @@ %0 - Unique key, if there is one. &l.canset DSF=or(gte(bittype(%#), 5), strmatch(%0, %vK)) @@ Find the first parent that does NOT hit the attr cap, or the parent that @@ has the attribute on it, if it already exists. @@ @@ Arg: @@ %0 - target object to parent @@ %1 - target attribute @@ Note: I don't like the naming convention here, but can't think of a better one. @@ If the name is too long (more than 399 characters on my game) it just gets @@ partially eaten. The objects are still there. I didn't implement a count @@ because I didn't want to call yet another function to get the number of @@ parents, since MUX doesn't have lparents(). The less recursion the better. &f.find-first-free-parent DSF= if( or( lt(attrcnt(%0), v(d.max-attributes)), hasattr(%0, %1) ), %0, if( t(parent(%0)), ulocal(f.find-first-free-parent, parent(%0), %1), null(strcat( setr(0, create(name(%0) - DSP %0, 10)), parent(%0, %q0), set(%q0, safe), tel(%q0, loc(%0)) ))%q0 ) ) @@ Set the data (global user function, should be wiz-only) @@ @@ Args: @@ %0 - target object @@ %1 - attr name:value (just like set()) @@ %2 - optional key, required for calling if not a wizard @@ Results: Nothing, just like set(). &f.global.setdata DSF= case(1, not(ulocal(l.canset, %2)), #-1 ERROR: HIGHER PERMISSIONS REQUIRED., gt(strlen(%1), v(d.max-attr-value-length)), #-2 ERROR: TOO MUCH DATA., not(strmatch(%1, *:*)), #-3 ERROR: INVALID DATA FORMAT. MUST BE NAME:VALUE., gt(strlen(first(%1, :)), v(d.max-attr-name-length)), #-4 ERROR: ATTRIBUTE NAME TOO LONG., if( or( t(setr(0, set(ulocal(f.find-first-free-parent, %0, first(%1, :)), %1))), eq(strlen(%q0), 0) ), @@(All good. Say nothing!), %q0 ) ) @@ All done! Anything past this point is gravy. @@ ----------------------------------------------------------------------------- @@ Set function up on #1 with: @@ @startup #1=@fo me=@function/privilege/preserve setdata=#DBREF_OF_DSF_GOES_HERE/f.global.setdata @@ @@ The function needs privilege to create its own objects and set data on things. @@ It needs preserve because we use %q0 in there and don't want to overwrite something else's %q0. @@ Because it's a privileged function, we need some other way to lock it down, thus the UNIQUE KEY thing. @@ Call function wherever you would call set(target, attrname:attrvalue) like so: @@ setdata(target, attrname:attrvalue)And some test code to use it...
@@ And here's a simple tester. All it does is let staff AND PLAYERS add data to @@ a specified object. Because it has its own unique key, it can do that. If you @@ want to restrict it to a specific bit type, @lock it. This is just a proof of @@ concept. @@ @@ Commands: @@ +setdata <attr>=<value> @@ +setdata/quiet <attr>=<value> @@ +getdata <attr> @@ +datastats @create Basic Data Storage Object <BDSO>=10 @set BDSO=safe @create Data Storage Command Tester <DSCT>=10 @set DSCT=safe inherit @fo me=&vD DSCT=[num(BDSO)] &vK DSCT=UNIQUE KEY HERE @@ Get all the data objects. @@ @@ Args: @@ %0 - target object @@ Results: Nothing, just like set(). &f.list-parents DSCT=if(t(parent(%0)), parent(%0) [u(f.list-parents, parent(%0))]) @@ Set the data loudly. @@ Format: +setdata attrname=attrvalue @@ Result: "Set." or error. &cmd-+setdata DSCT=$+setdata *=*:@pemit %#=[if(or(t(setr(0, setdata(%vD, %0:%1, %vK))), eq(strlen(%q0), 0)), Set., %q0)]; @@ Set the data quietly. @@ Format: +setdata attrname=attrvalue @@ Result: Nothing unless there's an error. &cmd-+setdata/quiet DSCT=$+setdata/quiet *=*:@switch [setr(0, setdata(%vD, %0:%1, %vK))]=,{},@pemit %#=%q0; @@ Get the data. @@ Format: +getdata attrname @@ Result: Data or a list of possible attributes it could be on using lattrp. &cmd-+getdata DSCT=$+getdata *:@pemit %#=[if(t(setr(0, xget(%vD, %0))), MSG: Your data is: %q0, MSG: Attribute not found. Did you mean: [lattrp(%vD/%0*)]?)]; @@ Show some data stats. @@ Format: +datastats @@ Result: The stats. &cmd-+datastats DSCT=$+datastats:@pemit %#=[repeat(-, 80)]%RBase object: %vD%RParents: [setr(P, [u(f.list-parents, %vD)])]%R[iter(%vD %qP, if(t(itext(0)), [repeat(-, 80)]%R[itext(0)] Name: [name(itext(0))]%RAttributes: [attrcnt(itext(0))]%RObject memory used: [objmem(itext(0))]%R),,@@)][repeat(-, 80)]I dunno how to solve the key issue, I have a headcold. You could solve just by making the function not-wizard, but then it fails at its primary requirement.
Throwing this up there in case someone besides me finds it useful and can solve the tricky bits.
-
-
Port to Rhost. <no, seriously>
-
-
I once created a system to handle ongoing, almost infinite logging, worked out the math you looked into, and immediately went to SQL to do the logging in a way that is easier to pull and harder to mess up. I may be able to find it somewhere, though last I remember I left it on The Reach, which means it may be on Fate's Harvest as the XP gain/spend log. I originally created it for a pose-logger system, in order to allow staff to keep an eye on the OOC Nexus so an always-on staff-bit didn't have to sit there.
The main question I have is: What is the core issue you're trying to solve?
Likewise is there any other way to achieve it? As fun as working out the numbers and making something to overcome the limitations can be (c'mon, this is what coders do for fun!), there are better ways for almost every situation.
And finally, your final statement makes it sound like you want to know how to limit something for non-wizards using the code. Can you clarify?
-
@ixokai said in TinyMUX: Info Storage:
@rook said in TinyMUX: Info Storage:
Port to Rhost. <no, seriously>
Use clusters.
With 32K LBUFS, at the absolute ceiling of 10,000 attributes per object, this would allow you with a @cluster of... hum... somewhere in the realm of an absolute maximum of 60 million attributes seen by a single entity.
For those not in the know, this is what @cluster is in RhostMUSH
@CLUSTER Command: @cluster[/<switch>] [[<arguments>][=<values>]] The @cluster command is used to 'cluster' or link together a set of 2 or more dbref#'s into a single entity to be used and accessed by a set of clustering commands and functions as the aforementioned single entity. This is used to allow more flexability of data storage, better organization and a way to allow better data handling of the database as a whole. For more help on what @cluster can do, please refer to the following switches available. Individidual help is available with 'help @cluster <switch>' Switch Description Switch Description --------- -------------------------- ----------- ----------------------- /new - start new cluster /add - add dbref# to a cluster /del - delete dbref# from cluster /clear - purge cluster /list - list cluster specifics /threshold - set threshold limit /action - set threshold action /edit - edit attr on cluster /set - set/unset attr on cluster /repair - repair cluster /grep - grep cluster for match /reaction - edit action on cluster /cut - 'cut' dbref# from cluster /trigger - trigger attr in cluster /func - Specify function action /regexp - use regexp pattern match /wipe - Wipe attrib(s) in cluster /owner - Owner check for /wipe /preserve - Preserve /wipe pattern See Also: clusters, cluster functions, cluster commands, > -
@ixokai said in TinyMUX: Info Storage:
@rook said in TinyMUX: Info Storage:
Port to Rhost. <no, seriously>
Use clusters.
MUX doesn't have Clusters. That is why I recommended that @Melpomene port their game to Rhost.
-
Clusters is like asking someone how to add paper to a spiral-bound notebook and getting a treatise on paper-making in the modern era.
Which is why I asked probing questions.
Sheesh. Some coders.
-
@rook said in TinyMUX: Info Storage:
@ixokai said in TinyMUX: Info Storage:
@rook said in TinyMUX: Info Storage:
Port to Rhost. <no, seriously>
Use clusters.
MUX doesn't have Clusters. That is why I recommended that @Melpomene port their game to Rhost.
I know? My pestering is partly why Ashen made clusters
-
Reason for code is essentially "because MUX doesn't have @clusters". This is sort of a polyfill for it - but only sort of, not exactly. I'd have to dig a lot deeper into everything clusters can do to make it a true polyfill.
I gave this code a lot of thought today and decided I wanted to do more with it. I'm working on a fuller version that supports indexing.
Why no SQL? Because I feel like sql(), a blocking call, is awkward and painful and non-native, and I've somewhat had it with "Your site's connection to the SQL server has dropped, please @restart!" messages. Fitting SQL's syntax, which I am fully fluent in, into softcode, hurts my soul. The right way to do it would be with @query, and well-written stored procedures, and honestly I felt that would be too much work for what I'm trying to do... but someone else could surely pull it off, and good luck to them.
The key thing. I'm probably having a brain fart over how the entire thing works, but I wasn't able to make the lock detect that a player calling it was calling it through an inherited-permissions object. What I want:
- Player can do +setdata command if it's not locked to wizards.
- Player can NOT do setdata(target, attr:value). (Not even on himself - because the player could end up with a bunch of parent objects. Honestly what we're restricting here is who the target can be, but I want that to be handled completely in code, without staffers having to manually, say, add the dbref of the newest storage object to some attribute somewhere...)
I fucked around with it and no matter how I arranged it, either both worked or neither did, so I said "fuck it" and went with the key thing.
What's this about indexing? I'm working on a stat system. Stats have titles, values, descriptions, tags, and, occasionally, stamps (for approval, date/time marking, etc). Essentially these items function like columns in a database with each new stat being a row. A row deserves an ID. IDs let you...
- Work with stats that have very similar names
- Identify disparate bits of data without worry that a typo will lead to bugs
- Stack categories in an attribute name like so: &_stat.3>2>47>5 JimBob=Cool gun stats here!
I've got it about to the point where it will store both indexed and non-indexed data reliably and distinguish safely between the two, and have added the appropriate getdata function for it (accepts index OR attribute name), but I feel like I need to kick it more solidly before I release it. I also need to finish writing test code for it.
I guess the answer to your question, @Thenomain, is... why would anyone need to store 25,000+ pieces of data (and not use SQL)? Coders, man.
But also because it's easy and fun. I'm sure there are other ways to solve it, but this one took three lines - create(), tel(), set(). Everything else is gravy. -
First: If you occasionally prod the SQL connection (say, once per minute), then MySQL itself will not time out the connection like it wants to. You would absolutely use @query, though I have never in my life seen sql() hang the game. This doesn't mean that it can't, but that I personally am willing to risk it. I'm a rebel.
Secondly: You say using SQL properly "too much work for what I want to do" and yet you're trying to create a cross-object data storage system. We all know the real reason is "why not?".
Thirdly: These are the reference shortcut that solves the 'who is calling this code?' issue:
%# - calling dbref %@ - referencing dbrefSo if you want someone to be able to run a +command and have that object have permissions, you'd do something like:
gte( bittype( %@ ), 5 ).Fourthly: For WoD I created something called 'stat path' that pretty much uses the attribute name as the index. I don't use lookup id technique because I want to code for non-coders too, so I have:
&_attribute.strength: 3 &_merit.language_(french): 1It is not the best technique, but I can kill a lot of birds with one stone, especially when I'm doing something such as deciding to search merits before gifts for partial name-matching 'sim'.
I don't think there's a wrong way to do it as long as your structure is normalized and consistent, but I wanted to say you aren't alone in your thinking.
-
@Thenomain Re: Second:
YUP. Hrm, though, I tried those substitutions and failed. Let me... kick it again when I do not have a headcold. I would like to get rid of that stupid key. Maybe I did something wrong in my fever-haze.
Yeah, the statpath idea was a good one, but I wanted shorter paths. Your comment on that Codebase Pot Pie thread about regexes in statpath searches, though, made me go "Hmm", and indexes was my idea of "maybe solve this". You still will have problems if you don't have delimiters, but... I think the lookups will be quicker. Maybe.
What I'm working on is an essentially universal base, eventually to be shoved under a pile of system-specific code. This is just one piece of the API for it. I'll see about getting it on github at some point when it's either doneish or abandoned so other people can play with it.
-
Delimiters don't make stuff slower.
Also, I have delimiters. The mighty and easier to type
., not that>is that much harder.As long as you add spaces after commas and semi-colons, you can do what you want.
Also: Add spaces after commas and semi-colons. Not only is it easier to read, but it's easier to keyboard navigate.
-
So, because I can't leave well enough alone, I designed a data-source and mostly data-type agnostic data storage module. It uses both the method above and the SQL method, because I'm a glutton for punishment (and honestly MySQL is interesting). It is here:
https://gist.github.com/ADHdev/66ccc9ac4a520da760045a456e3a301d#file-cheapassclusters-txt
That's the decompiled version, with the test object, plus a bunch of tests I used. I've tested it pretty thoroughly and am using it on the game I'm working on.
If anyone uses it, poke me here and let me know if you encounter problems with it. I'm curious to see if anyone else encounters the stored procedure issues I ran into, or if they're just issues on my own machine.
If people prefer it in a compiled format (just the code plskthx) let me know and I'll toss one up.
(Edit to update the permalink.)
-
If you make all the comments C-Type comments, then the whole text can be thrown at Muxify.
-
Meeeeeh, C-style comments... but I changed it, just for you. XD
-
https://gist.github.com/ADHdev/0f16992826cac3bcd00266f33dbceb87#file-basic-help - and here's a basic +help thing using it. It takes the basic data object and improves on it somewhat - fixed a few bugs, changed the syntax to be friendlier, more standard.
And yes, @Thenomain, C-style comments. XD
(Edit to update link to better permalink.)
-
@Melpomene
Your very last line makes a butt-ton usage of %b. Try replacing with [space(#)]? Just helps with readability. -
@melpomene said in TinyMUX: Info Storage:
Meeeeeh, C-style comments... but I changed it, just for you. XD
It saves everyone time. Instead of anyone having to ask you for the compressed code, they go there, select all, copy, go to Muxify, paste, done.
If you don't like it, don't use it. It's your github. You can structure the code in iambic pentameter for all I care.
Actually, I'd like to see that.
-
@rook I used https://www.orcpie.fun/muxwriter.html to turn it into that because I want to be able to grab "%b" on the wiki end and translate it into " ". (There will be <pre></pre> involved in this.) The idea here is to get something that doesn't involve wiki format translation - I'd rather translate MU* into wiki than vice versa. (We're using Dokuwiki, or at least we will be.) It would be a bit harder to parse text for "[space(\d+)]"; %b is a straight-up replace.
Should be pretty simple to th (the text) and see it on a MU* if you're curious - it's just a formatted version of the Commands list from the top.
-
Updated the +help sample to actually use MVC structure (sort of). Added proper help files for it (ha!). I would now consider the +help thing a full-fledged piece of code and no longer "basic". It's ready to be repurposed for whatever data storage needs to happen.
https://gist.github.com/ADHdev/0f16992826cac3bcd00266f33dbceb87#file-plushelp-txt
There have been some minor updates to Cheapass Clusters as I discovered it wasn't storing raw code, and fixed a few other bugs.
https://gist.github.com/ADHdev/66ccc9ac4a520da760045a456e3a301d#file-cheapassclusters-txt
Not sure anyone's using these, but if you are, you probably ought to update.
And for my next trick: anvils. ...or rather, a generic stat system.
(Yes, so you non-WoDers can have a chargen that isn't locked to that system.) This will probably take some time, though. I'll start a new topic when ready.
