The guy where I can only shake my head when I see his code, and he is really proud of if implementations, while he

- doesn't care about warnings

- breaks builds and doesn't care

- doesn't care about code styles and indents in a very column based way

- adds tons of comments to his code, mostly hard to understand, and sometimes that much you can hardly find the code

- implements a tokenizer where you have to inherit from its interface (Why would I wanna implement whole functions for a tokenizer and not just use it in place where needed? How do I use two of those in one class?)

- implement a "generic" state machine base class with fixed lengths array of 3 events and 3 strings (Why would I need events and strings hardcoded in a "generic" state machine? Why a maximum of 3?)

- once delivered a software without the needed runtime components, so the whole system (embedded device) wasn't working properly and only by chance missed the point of disabling update mechanisms

- make your ears bleed about his big inventions whenever he sees you, no matter how often he already told you about that blazing new feature

A dev team has been spending the past couple of weeks working on a 'generic rule engine' to validate a marketing process. The “Buy 5, get 10% off” kind of promotions.

The UI has all the great bits, drop-downs, various data lookups, etc etc..

What the dev is storing the database is the actual string representation FieldA=“Buy 5, get 10% off” that is “built” from the UI.

Might be OK, but now they want to apply that string to an actual order. Extract ‘5’, the word ‘Buy’ to apply to the purchase quantity rule, ‘10%’ and the word ‘off’ to subtract from the total.

Dev asked me:

Dev: “How can I use reflection to parse the string and determine what are integers, decimals, and percents?”
Me: “That sounds complicated. Why would you do that?”
Dev: “It’s only a string. Parsing it was easy. First we need to know how to extract numbers and be able to compare them.”
Me: “I’ve seen the data structures, wouldn’t it be easier to serialize the objects to JSON and store the string in the database? When you deserialize, you won’t have to parse or do any kind of reflection. You should try to keep the rule behavior as simple as possible. Developing your own tokenizer that relies on reflection and hoping the UI doesn’t change isn’t going to be reliable.”
Dev: “Tokens!...yea…tokens…that’s what we want. I’ll come up with a tokenizing algorithm that can utilize recursion and reflection to extract all the comparable data structures.”
Me: “Wow…uh…no, don’t do that. The UI already has to map the data, just make it easy on yourself and serialize that object. It’s like one line of code to serialize and deserialize.”
Dev: “I don’t know…sounds like magic. Using tokens seems like the more straightforward O-O approach. Thanks anyway.”

I probably getting too old to keep up with these kids, I have no idea what the frack he was talking about. Not sure if they are too smart or I’m too stupid/lazy. Either way, I keeping my name as far away from that project as possible.

Definitely Rust, and a bit Haskell.

Rust has made me much more conscious of data ownership through a program, to the point that any C/C++ function I wrote that takes a pointer nowadays gets a comment on ownership.

I wish it was a bit less pedantic about generics sometimes, which is why I've started working on a "less pedantic rust", where generics are done through multiple dispatch à la Julia, but still monomorphising everything I can. I've only started this week, but I already have a tokenizer and most of the type inference system (an SLD tree) ready. Next up is the borrow checker and parsing the tokenized input to whatever the type inference and borrow checker need to work with, and of course actual code generation...

Haskell is my first FP language, and introduced me to some FP patterns which, turns out, are super useful even with less FP languages.

"The word ‘tokenizer’ makes a lot more sense, but ‘lexer’ is so much fun to say that I use it anyway."

No wonders people think a lot of programming subjects are intimidating.

Last night, while under the full belief that I could write a very simple lisp interpreter, I was awake until stupid o'clock and to my credit I got the tokenizer working and produces an output of parsed code. It's really basic but I was pleased to have gotten anything correctly parsed at that time.

But I'm also sitting outside my apartment waiting for a locksmith because lack of sleep left me unprepared to function correctly today and I'm now locked out...

Well done me!

I've made a NLP tokenizer which is 30 times faster than clearNlp and standford tokenizer, last year. It is rich and require very less space(don't load any model). Should i make it open source?

Writing tokenizer for a new language.

Started writing a parser for moonscript. Because I want to do my own syntax highlighting and error support.

I'm sorry, but was this supposed to difficult? Every article I read claimed this was gonna be some impossible feat of herculean effort. I half dreaded it, the other half was kinda elated.

Only it didnt live up to the hype. The tokenizer is a glorified character stream. The lexer is little more than a tokenizer, and the "most complicated" bit is nothing but a fancy transformation of the token output into a tree.

I'm completely to new parsers proper and semantic checking and maybe that's why it seemed easy, but I dont see what all the forewarning in tutorials were ever about.

My tokenizer can how tokenize many statements rather than just a single statement. ^_^ I'll make it open source once I've done a bit more work on it but so far I'm happy.

! wk95

My project back in university, where I used bash and NLP and Python to create a utility thay would execute sentences written in English. Much like typing "change my wallpaper to abc.jpg"

Even though the tokenizer took almost five minutes to tokenize a sentence ( longer than five words ), and the parser took even longer, I still love it, for it was my first dive into ML !

Anyone ever struggle with tokenizers? I'm writing one by hand and it's frustrating mr so much, because at times I think my logic gets flawed somewhere inside a good loop and things get slipped or missed I don't know how to explain it.

Question for someone who uses Mongo Atlas Search:

If I'm only interested in autocomplete from the start of the text, which is more performant?

1) standard analyzer + edgeGram tokenizer

2) keyword analyzer + edgeGram tokenizer

I don't see why I should index separate words if I don't care about random positions :/

Thank you