Information Storage Question

NightAngel12

For those that have more experience with soft code, what would you say is the maximum amount of information that a single attribute can contain?

To give some context for this question to help coders with answering me:

&infodump ####=Text|dbref#.DESC<INT@INT>INT

get(after(before(after(before(<CODE>)))))))))

sort of storage system? It's doable and with enough patience it works fine for certain things. I'm curious just how much information can be shoved into one of these attributes. Can I repeat special characters? Do certain characters not work? etc.

ixokai

About 8k, though there' some overhead. (On Rhost this is a compile time question; from the default of 4k up to 64k (not recommended))

Also check out the 'grab' function if you have structured data in an attribute.

grab(get(object/attribute),field:*,|)

Would quickly retrieve 'field:<data>' from field:data|field2:data|field3:data.

The only 'special characters' are what you choose to be delimiters. I use delimiters like '->' and '|' to maximize how much I can put into it without possibly using something a user would want to fill out.

NightAngel12

@ixokai Thank you! That really helps me out with something I've been tinkering with on one of my many, many, backburners.

Thenomain

If a field is going to hold a text dump, I try to make sure that either there is a separate attribute or that it's the last field. This usually works for me, unless I know the system is going to be used heavily enough that number of attributes per object or amount of code around pulling the base attribute name are going to be at a premium.

I tend to one-character delimiters, mostly because lazy but also because there are a few (maybe two) functions in TinyMUX that can handle only one-character delimiters. ` and | are my favorites.

Remember to scrub or replace delimiter characters from user-entered text before adding them to the attribute.

Mercutio

For your consideration:

&reg obj=(?P<Text>.+?)|(?P<dbref>#\d+)\.(?P<desc>.+?)<(?P<int1>.+?)@(?P<int2>.+?)>(?P<int3>.+)
th regedit(stuff,v(reg),r(dbref))

Or:

regmatch(stuff,v(reg),text dbref desc int1 int2 int3) %q<int1>

However, I might recommend you don't store like that, if possible, and stick them into seperate attributes... or in SQL if you are going to store a lot of data. You've got roughly 2000 attributes to play with on for instance, Pennmush, per object.

Example:

object/stats`abilities`intelligence

If you want to check what your max length is to stick into something:

th strlen(repeat(-,234234234))

Note however, that max command length is lower than that. Half, in fact, for PennMUSH.

Thenomain

@Mercutio said in Information Storage Question:

For your consideration:

&reg obj=(?P<Text>.+?)|(?P<dbref>#\d+)\.(?P<desc>.+?)<(?P<int1>.+?)@(?P<int2>.+?)>(?P<int3>.+)
th regedit(stuff,v(reg),r(dbref))

Important Note: Mux does not have regedit. (Dammit.)

If you want to check what your max length is to stick into something:
th strlen(repeat(-,234234234))
Note however, that max command length is lower than that. Half, in fact, for PennMUSH.

Will this fill out the rest?

@fo me=&test object=[repeat(-,234234234)]
@fo me=@edit object/test=$, [repeat(-,234234234)]
think strlen( get( object/test ))

Mercutio

@Thenomain said in Information Storage Question:

@fo me=@edit object/test=$, [repeat(-,234234234)]

I wish. it has to do with the player's command length. So if I recall correctly, the only way you will get it is by copying the output of [repeat(...)] and then saving that into an attribute. It generally just comes up if you try to store... for instance... poses. Or deal with bots.

Ashen-Shugar

@Mercutio said in Information Storage Question:

@Thenomain said in Information Storage Question:

@fo me=@edit object/test=$, [repeat(-,234234234)]

I wish. it has to do with the player's command length. So if I recall correctly, the only way you will get it is by copying the output of [repeat(...)] and then saving that into an attribute. It generally just comes up if you try to store... for instance... poses. Or deal with bots.

MUX and Rhost allow:

think config(lbuf_size)

It will return the size of the buffer on the particular game.
Not sure if Penn has an equiv.

Thenomain

@Ashen-Shugar

I don't know why anyone would suggest not using 64k buffers. I can't code without at least 16 anymore, or at least SQL which is preferable but not universal.

Ashen-Shugar

@Thenomain said in Information Storage Question:

@Ashen-Shugar

I don't know why anyone would suggest not using 64k buffers. I can't code without at least 16 anymore, or at least SQL which is preferable but not universal.

It's not so much that mush can't handle 64K buffers or more, it's that older routing and TCP buffers have issues with it.

Some systems cap TCP buffers at 32K which can cause a retry and a retransmit on packets over that 32K, which depending on your network latency and reliability could potentially mean lost packets (seemingly information being cut off).

Sometimes this is in relation to the receive window size being set to a ceiling of 32K, so any packets over 32K requires an acknowledgement before sending the remainder of packets. In around-robin TCP single-threaded engine, this could cause potential of issues and would likely require some back-end handling for additional buffering of packets not sent. Theory, as I've not researched it overly much on send/receive TCP window size alterations.

When I played with 64K lbufs it seemed to work fine with people with fat network systems but those who were on older ISP's and slower networks occasionally complained about information being cut off.

So it's why I usually suggest not going over 32K lbufs.

Mercutio

As a sidenote, it's really easy to set Penn to 16k. But it can't go above that. [Edit mushtype.h]

ixokai

@Thenomain said in Information Storage Question:

@Ashen-Shugar

I don't know why anyone would suggest not using 64k buffers. I can't code without at least 16 anymore, or at least SQL which is preferable but not universal.

I'm seriously curious what the heck you're doing in MUSH that a 32k lbuf isn't considered overkill.

Thenomain

@ixokai said in Information Storage Question:

@Thenomain said in Information Storage Question:

@Ashen-Shugar

I don't know why anyone would suggest not using 64k buffers. I can't code without at least 16 anymore, or at least SQL which is preferable but not universal.

I'm seriously curious what the heck you're doing in MUSH that a 32k lbuf isn't considered overkill.

Then you haven’t ever seen Reach/Fallcoast/Fates Harvest’s stat system, you lucky bastard.

Mercutio

@Thenomain
Sounds like a good case for using @dolist instead of one giant @pemit.

ZekeTheG33k

@Mercutio said in Information Storage Question:

@Thenomain
Sounds like a good case for using @dolist instead of one giant @pemit.

Or a series of @includes to break that up.

Cheesegrater

@Ashen-Shugar said in Information Storage Question:

Some systems cap TCP buffers at 32K which can cause a retry and a retransmit on packets over that 32K, which depending on your network latency and reliability could potentially mean lost packets (seemingly information being cut off).

That's not how TCP works. Packets will be retransmitted more or less forever. The connection would drop completely if a packet was undeliverable or reached a hard timeout. The worst thing that repeated retransmits would cause is some lag.

Message sizes don't matter because the OS's underlying implementation is responsible for splitting oversize packets anyway.

Thenomain

@Mercutio said in Information Storage Question:

@Thenomain
Sounds like a good case for using @dolist instead of one giant @pemit.

The lbuf does more than give you an output limit; it's also the maximum amount of text that the system can process at a time. For, say, lists of attribute names in searches.

Mercutio

@Thenomain
You can make use of nlsearch() combined with an upper and lower bound to solve that. A bit of a pain in the rear. nattr() with xattr() for the attribute lists.

But if you're storing such large amounts... SQL.

Thenomain

@Mercutio said in Information Storage Question:

@Thenomain
You can make use of nlsearch()

<tinymux>What's nlsearch()</tinymux>

--

edit: You don't have to tell me that there are better ways to do it. I do, tho, have remind people like us who know a shit-ton of techniques in this hobby (and that's sad in and of itself) that not everyone does.

Mercutio

@Thenomain
Certainly. That's why I stated them. I'm fairly certain you do (though not 100% of course - I am never 100% certain of such things).

P.S. God-damnit, MUX. Get your act together.
P.P.S. I used to actually do these: http://community.pennmush.org/node/924 -- in order to introduce problem concepts in PennMUSH and potential solutions.