Belovèse tool for extracting data from EDU

[Belovèse]

Welcome everyone,

Here is the link to the new thread for this tool I've posted the new version, that can also export EDA and EDCT

Please post here any bug report, feedback and suggestions.

How to use:
- Download and unzip.
- Keep the folder as it is: for the program to work you must have EDU_to_CSV.exe, EDU_to_CSV.mxf and subfolder "resources" in the same folder
- Run EDU_to_CSV.exe (not .mxf)
- Select a file, eventualy select what parameters you want to see, then click save and save in .csv format.
- Finally open the .csv file with a spreadsheet editor (Excel, LibreCalc, what you like)
- If asked, use TAB as separator for the import to a spreadsheet.
- enjoy!


Changelog

Spoiler Alert, click show to read: 

version 0.31
- fixed voice_type bug
- fixed "soldier health" bug
- fixed hardness bug

version 0.3
- added a "STOP" button
- workaround for the changing number of arguments in stat_pri
- corrected "law": it was upside down (is_peasant was giving 100% law instead of 50%)
- added almost all single columns for the supported entries

version 0.2
- reworked pretty much everything under the hood!
- added progression more feedback
- minor graphical improvement
- is_peasant is in a different column, to show the civil order impact (50% or 100%
- a lot of the stats have their own columns now
- blank line bug is fixed

Version 0.1:
- added support for "SSHIP_class" separator in the "dictionary" entry

Here's a vague roadmap

Spoiler Alert, click show to read: 

- Release v0.4: july
- Adding missing entries (era, officer, formation)
- Adding persistent option (remenber your last selections
- adding group selection (to select/unselect a whole entry, not just all parameter.)
- under-the-hood tweaks and cleanup
- Adding support for EDA and EDCT: after that
- After that I'll open a workshop tread and will add support for other mods, on demand and if I can!

Here is also a tool to parse EDU to export_unit.txt (to add more info on unit cards), provided by HaruHalas and based on a script by Cryoshakespear.
Instructions:

Spoiler Alert, click show to read: 

1. To use it create a directory say "test", inside move the script, copy exports_units.txt and and export_descr_units.txt there.
2. Open the current export_units.txt file and click on the Encoding tab at the top of Notepad++. It should indicate it's encoded in UCS-2 LE BOM. Click on "Encode in UTF-8". Save the file.
3. Run the script (using console do "py EUD_Parsing_Script.py" It will generate a new file named [EUD_Mod]export_units.txt.
4. Open the file [EUD_Mod]export_units.txt - select all the text (Ctrl-A) and *copy* it manually (yes, just do this, trust me).
5. Go to data\text\export_units, open it, delete and replace all the text in it with the text you've copied from the [EUD_Mod]export_units.txt file. remember to have a backup available.

Original post:

Spoiler Alert, click show to read: 

Quick question for the modding team: do you use the notepad++ syntax highlighting made by GrnEyedDvl, and if yes would you be interested if I were to update it?

Long story short, I'm trying to understand how modding the game works by going through the forum and the files, but I looked for highlighting (since it was a pain to read without!) and only found GrnEyedDvl's syntax with active links.
I played around with the syntax and managed a few updates, like: dark mode, proper code folding, more colours and better keyword recognition (for exemple it was wrongly highlighting "Trait" inside names like: "SomethingTraitOtherthing").

It's a work inprogress, but f it would be usefull for you I would put more time on it.

[Jurand of Cracow]

I don't use it, frankly speaking. You may have a look at the script.txt for 097, I think I've made some order and put a few hints for the next modders.

What I need the most is a program that would transform EDU (export_descr_units.txt) into a spreadsheet file so that I can get something like this (this one is for 092):

[Belovèse]

Jurand, I tried a few things and could make something like this: https://we.tl/t-rwled9oEKW (it's a excel file)

It's a first draft, and I did not went through the whole file to search for mistakes: I would like to know first if it's what you need, before putting more work on it.

A few limitations: I can't use the stats that are in two words (like "banner holy") or are on two lines (like "officer")

[Jurand of Cracow]

Jurand, I tried a few things and could make something like this: https://we.tl/t-rwled9oEKW (it's a excel file)

It's a first draft, and I did not went through the whole file to search for mistakes: I would like to know first if it's what you need, before putting more work on it.

A few limitations: I can't use the stats that are in two words (like "banner holy") or are on two lines (like "officer")

Hi Beleovese, this is what I was looking for. It will make my life easier, for sure.
Is it possible to have in a separate table:
- the text that is after ; in the entry: dictionary (there're in-game names and types of the units, so it's an important entry)
- the numbers in the "soldier" entry (so that there're two columns: one for soldier, the other for numbers (it's important as the first of the numbers signifies the size of the unit)
- in "stat_pri" the first two numbers in one column, and the rest in another column
- in "stat_sec" similarly, but if the numbers are "0 0" then the it'd be left empty
- "stat_pri_armor" - also numbers and text separated
- "stat_ground" - the best would be if each number would be in the separate column
- "stat_mental" - each should be in the separate column (morale, discipline, training)
- "stat_cost" - first three numbers should be in the separate columns,

[Belovèse]

I'll give it a try tomorow, to see what I can do. If it work I'll post the program. In the dictionnay entry I think I'll have trouble separating data, so it'll likely be a single column for that.
Since last post I got curious and managed to integrate the banner and era entries in the program.
What I'm writing is not very efficient, as I only know one programming software that is designed for live music! But hey, if it work's, it ain't stupid, or so they say.

[Kilo11]

I would also be interested in hearing how this ends up working out. I myself am not super gifted with crafting such programs, but a friend of mine has fun making crawlers and bots to glean information from certain websites (nothing sinister; he uses it to help him figure out what kind of fishing rod to buy, and like that, by comparing prices and models across multiple platforms). But he is good to making rather simple things that are good at collecting and crunching info from tabular environments (the structure of the EDB makes it likely that he could do something for that as well). So yeah, I'll keep an eye on this.

[Jurand of Cracow]

In the dictionnay entry I think I'll have trouble separating data, so it'll likely be a single column for that.

In this entry there's always ; used, I mean: each and every unit has it (or should have it). Given this "no exception" rule, I think that it could be not so difficult.

[Belovèse]

Here is what you asked, Jurand, I think: https://we.tl/t-ZLVFyGVjfU
If it's what you need, I just have to make the interface more understandable and a few tweak, and I could post it with a mega or dropbox link
I'll make it so you can choose the stats you want to export. If you point me to the right documentation I'll use better names for the columns also. If most of the mods use the same EDU structure, I'll post it in the worshop I think

In this entry there's always ; used, I mean: each and every unit has it (or should have it). Given this "no exception" rule, I think that it could be not so difficult.

indeed you were right!

I would also be interested in hearing how this ends up working out. I myself am not super gifted with crafting such programs, but a friend of mine has fun making crawlers and bots to glean information from certain websites (nothing sinister; he uses it to help him figure out what kind of fishing rod to buy, and like that, by comparing prices and models across multiple platforms). But he is good to making rather simple things that are good at collecting and crunching info from tabular environments (the structure of the EDB makes it likely that he could do something for that as well). So yeah, I'll keep an eye on this.

I also would like to learn how to program like your friend, I think he could make something lighter and more powerfull! Someday...

[Jurand of Cracow]

That's great, @Belovese!
I think it's very difficult to split the "dictionary" so that the "type" (feudal knight, urban militia...) are in a different column? What I'd like to have is possibility to group the units by type - to compare them to see discrepancies etc.
It's very close, would it be possible to have:
- stat_pri numbers (and stat_sec as well) in two different columns?
- stat_pri armour in three columns (this is very important: they describe quite different characteristics of the unit),

[Belovèse]

My bad, I used the same link twice and can't edit my posts yet, here is the program: https://www.dropbox.com/s/bx1pf4ez01...o_CSV.zip?dl=0

[Jurand of Cracow]

@Jurand and @Kilo, here is the program: https://www.dropbox.com/s/c1rusr30hp...space.txt?dl=0

I could do what you asked, except for the "dictionary" entry. The names and types of units are not always the same number of words, and I could not find a way around that. Best solution I can think of is to use the filter options in Excel (like I did in the file included), sorry!

There is a small bug, caused by empty lines inside og a unit stats. In the file you sent me Jurand, it was in Druhzina, Mutatawwi and Sipahi. Here is a file without these lines: https://www.dropbox.com/s/c1rusr30hp...space.txt?dl=0

Would it be possible for you to make this separate entry if we'd put in the dictionary line for each of the units an additional " ; " (or any other character) between the name and the type?
The empty lines will be removed.

[Belovèse]

Would it be possible for you to make this separate entry if we'd put in the dictionary line for each of the units an additional " ; " (or any other character) between the name and the type?
The empty lines will be removed.

I'll give it a try first, I'm really not sure so I have to test. Howerver, It fells like reworking the entire EDU is maybe more work than just using the "filter" option in Excel. But if it'll help you I will try of course!

Asides that, does it work as expected? I forgot to replace 0 0 by --- in stat_sec, and I will also search for what is exactly is each number to give the columns better names, more descriptive.

[Jurand of Cracow]

yes, it does, it's already a great tool for me to compare units.

on the column titles you may have a look at the SS6.4 units data file

on the dictionary - I think I can make very easily (search ", feudal levy", replace with " ; feudal levy" will do for 107 units, and other types as well, then only looking for exceptions)

[Belovèse]

Would it be possible for you to make this separate entry if we'd put in the dictionary line for each of the units an additional " ; " (or any other character) between the name and the type?
The empty lines will be removed.

I'll give it a try first, I'm really not sure so I have to test. Howerver, It fells like reworking the entire EDU is maybe more work than just using the "filter" option in Excel. But if it'll help you I will try of course!

Asides that, does it work as expected? I forgot to replace 0 0 by --- in stat_sec, and I will also search for what is exactly is each number to give the columns better names, more descriptive.

[Jurand of Cracow]

Hi Belovese,
your program does work as expected. Here is the EDU without empty lines (those 3 cases), and also with " ; " in dictionary line for each unit separating a very important info for us: our type of unit.
If you could add the column for that additional info, and separate two numbers in the stat_sec, this would be great!cheers
JoC

[Belovèse]

Hi Belovese,
your program does work as expected. Here is the EDU without empty lines (those 3 cases), and also with " ; " in dictionary line for each unit separating a very important info for us: our type of unit.
If you could add the column for that additional info, and separate two numbers in the stat_sec, this would be great!cheers
JoC

Happy to hear it's working well! I followed your idea and can indeed make separate entries for "dictionary", but can't use punctuation (because of how the program I use works). So I will used a word instead, and since "type" is already an entry is the EDU, I will use "class" instead if you agree. I will update the program and the EDU you sent me, sometime today I think.

If you have other improvement ideas I'll be happy to hear them!

[Belovèse]

Happy to hear it's working well! I followed your idea and can indeed make separate entries for "dictionary", but can't use punctuation (because of how the program I use works). So I will used a word instead, and since "type" is already an entry is the EDU, I will use "class" instead if you agree. I will update the program and the EDU you sent me, sometime today I think.

If you have other improvement ideas I'll be happy to hear them!

Hey Jurand, I tried to implement the dictionary entry but it kind of messed the program (or I did something, don't know what!). I'll continue to work on it but can I open another thread to easily post updates and stop spamming the general discussion thread?

[Jurand of Cracow]

Happy to hear it's working well! I followed your idea and can indeed make separate entries for "dictionary", but can't use punctuation (because of how the program I use works). So I will used a word instead, and since "type" is already an entry is the EDU, I will use "class" instead if you agree. I will update the program and the EDU you sent me, sometime today I think.

If you have other improvement ideas I'll be happy to hear them!

Class is also an entry in the EDU, and actually it's pertinent (light, heavy, missile or spearmen).
On the other hand, "type" is an entry, but it has not relation to the concept of "type" (many similar units), it simply an internal code name of the unit (one unique name).
So I'd stick to the "type" name of they types, but I think I'd tinker a bit with the types (discussion on this in another thread).

I'm looking forward to that programm from you, already the current enabled me to spot many things in the EDU.

[Belovèse]

Class is also an entry in the EDU, and actually it's pertinent (light, heavy, missile or spearmen).
On the other hand, "type" is an entry, but it has not relation to the concept of "type" (many similar units), it simply an internal code name of the unit (one unique name).
So I'd stick to the "type" name of they types, but I think I'd tinker a bit with the types (discussion on this in another thread).

I'm looking forward to that programm from you, already the current enabled me to spot many things in the EDU.

I figured that halfway through changing the EDU you sent me... so I went for "SSHIP_class" as keyword, and here is the EDU based on the last you sent me. I hope you haven't yet made any changes!

About the program I went for a full rework, but it should more flexible and easier for me to add new functionalities, I think it'll take me a few days.

Thank you for opening the thread, as soon as I can edit my own posts I'll clean the first one and will put everything in the same place.

[Jurand of Cracow]

I figured that halfway through changing the EDU you sent me... so I went for "SSHIP_class" as keyword, and here is the EDU based on the last you sent me. I hope you haven't yet made any changes!

Well, as a matter of fact, we did. I didn't know you'd be making changes in the EDU. There're some accents changed by JLMP; I've also run your program, had a look at some units, and adjusted many plances (eg. recruitment times of urban militia, or some heat stats, and also added type "artillery" to the relevant units). Your program is very helpful in finding odd things.

EDU improved version is here.

However, I think it's easy for you to make your changes: just Ctrl-F " ; " and replace with your text, and you'll get 484 cases (so 10 units missing, to be found otherwise).
But I would be inclined to use in the EDU something more simple, why to make long < dictionary Javelinmen ; Levy Javelinmen SSHIP_class feudal levy >, and not simply < dictionary Javelinmen ; Levy Javelinmen ; feudal levy > ?

And just drop a line before if I shouldn't meddle with EDU ;-)