Feat/node insertion by DieracDelta · Pull Request #17 · DieracDelta/flake_generator

DieracDelta · 2021-05-02T20:29:27Z

This is a draft PR for AST node insertion. Objectives:

create AST nodes for attribute sets
convert between Input type and attribute set
clean code
- so many places need &str instead of String
- better file naming to match rest of project
- clippy
tie into main control loop so we can create inputs
more tests

Resolves #18 and #14 .

fogti · 2021-05-02T22:00:25Z

+    GreenNode::new(kind, children)
+}
+
+pub fn gen_attr_set(attr_pairs: Vec<(String, String)>) -> GreenNode{


Suggested change

pub fn gen_attr_set(attr_pairs: Vec<(String, String)>) -> GreenNode{

pub fn gen_attr_set(attr_pairs: &[(String, String)]) -> GreenNode{

I'm trying to do this, and running into issues here. It looks like there's not an easy way to convert from a vector to an array..

DieracDelta · 2021-05-10T00:23:54Z

Input generation works, and I've added in a nixpkgs_fmt pass to make sure things look nice. I've added a couple of tests as well. Code quality is poor; lots of unused imports, dead code, etc. A bunch of the processing done in main.rs does not belong. Inputs_utils needs to be split out into something more modular that matches with the rest of the style. Not ready to merge, but getting there.

I also think that a bunch of the functions in utils.rs and input_utils.rs morally speaking belong in rnix. I'm just filling in the gaps done by TODOs. I was hoping for more discussion on the issue I opened there...

DieracDelta · 2021-05-10T01:15:19Z

Also, inputs need to be (1) checked, and (2) there should be some sort of type generation. Ideally, we should support other types of flake inputs that covers the entire spec. Path/git/, enabling submodules, revisions, etc. Essentially, everything specified here. I think I'd prefer to do these in separate PRs though. The scaffolding here can define the basis for further work.

fogti · 2021-05-12T13:07:52Z

+    let kind = NixLanguage::kind_to_raw(NODE_KEY_VALUE);
+    let assign_kind = NixLanguage::kind_to_raw(TOKEN_ASSIGN);
+    let whitespace_kind = NixLanguage::kind_to_raw(TOKEN_WHITESPACE);
+    let semicolon_kind = NixLanguage::kind_to_raw(TOKEN_SEMICOLON);
+    let children = vec![
+        NodeOrToken::Node(key),
+        NodeOrToken::Token(GreenToken::new(whitespace_kind, " ")),
+        NodeOrToken::Token(GreenToken::new(assign_kind, "=")),
+        NodeOrToken::Token(GreenToken::new(whitespace_kind, " ")),
+        NodeOrToken::Node(value),
+        NodeOrToken::Token(GreenToken::new(semicolon_kind, ";")),
+        NodeOrToken::Token(GreenToken::new(whitespace_kind, "\n")),
+    ];


Suggested change

let kind = NixLanguage::kind_to_raw(NODE_KEY_VALUE);

let assign_kind = NixLanguage::kind_to_raw(TOKEN_ASSIGN);

let whitespace_kind = NixLanguage::kind_to_raw(TOKEN_WHITESPACE);

let semicolon_kind = NixLanguage::kind_to_raw(TOKEN_SEMICOLON);

let children = vec![

NodeOrToken::Node(key),

NodeOrToken::Token(GreenToken::new(whitespace_kind, " ")),

NodeOrToken::Token(GreenToken::new(assign_kind, "=")),

NodeOrToken::Token(GreenToken::new(whitespace_kind, " ")),

NodeOrToken::Node(value),

NodeOrToken::Token(GreenToken::new(semicolon_kind, ";")),

NodeOrToken::Token(GreenToken::new(whitespace_kind, "\n")),

];

use NixLanguage::kind_to_raw;

use NodeOrToken::{Node, Token};

let kind = kind_to_raw(NODE_KEY_VALUE);

let assign_kind = kind_to_raw(TOKEN_ASSIGN);

let whitespace_kind = kind_to_raw(TOKEN_WHITESPACE);

let semicolon_kind = kind_to_raw(TOKEN_SEMICOLON);

let children = vec![

Node(key),

Token(GreenToken::new(whitespace_kind, " ")),

Token(GreenToken::new(assign_kind, "=")),

Token(GreenToken::new(whitespace_kind, " ")),

Node(value),

Token(GreenToken::new(semicolon_kind, ";")),

Token(GreenToken::new(whitespace_kind, "\n")),

];

Can't import NixLanguage::kind_to_raw. Not sure why....

Oh... (that might be the case because the method is defined in a trait impl)
Maybe create a small wrapper for it then:

#[inline(always)] fn kind_to_raw(x: rnix::SyntaxKind) -> rowan::SyntaxKind { NixLanguage::kind_to_raw(x) }

fogti · 2021-05-16T14:54:34Z

 use rowan::{api::SyntaxNode, GreenNode, GreenNodeBuilder, GreenToken, Language, NodeOrToken};
 use std::string::ToString;
+use NodeOrToken::{Node, Token};


Suggested change

use rowan::{api::SyntaxNode, GreenNode, GreenNodeBuilder, GreenToken, Language, NodeOrToken};

use std::string::ToString;

use NodeOrToken::{Node, Token};

use rowan::{api::SyntaxNode, GreenNode, GreenNodeBuilder, GreenToken, Language, NodeOrToken::{self, Node, Token}};

use std::string::ToString;

fogti · 2021-05-16T14:57:45Z

    let close_curly_kind = NixLanguage::kind_to_raw(TOKEN_CURLY_B_CLOSE);
    let attr_set_kind = NixLanguage::kind_to_raw(NODE_ATTR_SET);
    let whitespace_kind = NixLanguage::kind_to_raw(TOKEN_WHITESPACE);
    let mut token_vec = Vec::new();


suggestion: replace the complete following stuff with this:

let mut token_vec = Vec::new(); let mut tokens = vec![ Token(GreenToken::new(open_curly_kind, "{")), Token(GreenToken::new(whitespace_kind, "\n")), ]; tokens.extend(attr_pairs .into_iter() .map(move |(k, v)| Node(new_key_value(k, v)))); tokens.push(Token(GreenToken::new(close_curly_kind, "}")));

fogti · 2021-05-16T15:01:14Z

+        let kind: rowan::SyntaxKind = NixLanguage::kind_to_raw(NODE_STRING);
+        node.start_node(kind);
+        let start_string_kind: rowan::SyntaxKind = NixLanguage::kind_to_raw(TOKEN_STRING_START);
+        node.token(start_string_kind, "\"");
+        let string_content: rowan::SyntaxKind = NixLanguage::kind_to_raw(TOKEN_STRING_CONTENT);
+        node.token(string_content, &item.val.to_string());
+        let end_string_kind: rowan::SyntaxKind = NixLanguage::kind_to_raw(TOKEN_STRING_END);
+        node.token(end_string_kind, "\"");


Suggested change

let kind: rowan::SyntaxKind = NixLanguage::kind_to_raw(NODE_STRING);

node.start_node(kind);

let start_string_kind: rowan::SyntaxKind = NixLanguage::kind_to_raw(TOKEN_STRING_START);

node.token(start_string_kind, "\"");

let string_content: rowan::SyntaxKind = NixLanguage::kind_to_raw(TOKEN_STRING_CONTENT);

node.token(string_content, &item.val.to_string());

let end_string_kind: rowan::SyntaxKind = NixLanguage::kind_to_raw(TOKEN_STRING_END);

node.token(end_string_kind, "\"");

node.start_node(NixLanguage::kind_to_raw(NODE_STRING));

node.token(NixLanguage::kind_to_raw(TOKEN_STRING_START), "\"");

node.token(NixLanguage::kind_to_raw(TOKEN_STRING_CONTENT), &item.val.to_string());

node.token(NixLanguage::kind_to_raw(TOKEN_STRING_END), "\"");

fogti · 2021-05-16T15:02:20Z

+pub(crate) struct Key {
+    val: SmlStr,
+}


I think a simple newtype struct would be a better fit, unless you intend to add more fields.

Suggested change

pub(crate) struct Key {

val: SmlStr,

}

pub(crate) struct Key(SmlStr);

DieracDelta · 2021-06-13T22:22:20Z

@zseri sorry I haven't pulled in your changes yet. I think that in order for this to work in a scalable manner (we'd like to pull in some sort of evaluation mechanism so we can meaningfully make edits and check their correctness), the IR needs to be a bit more flushed out. Going to focus on that for a bit then cleanup in the input code. In essence we're making our own IR here. I think it should at a higher level than the output of the parser, and we should have some sort of mechanism to switch between IR and GreenNodes. We'll parse the structure into this higher level IR, make changes based on user request, then push down to GreenNodes again.

fogti

I would suggest splitting the EDSL-like + conversion between that and rnix, rowan into a separate crate (maybe rnix-ddhir), so we can lay out a cleaner interface for it rather sooner than later.

fogti · 2021-06-14T19:33:09Z

+    String,
+    Function,
+    List(Box<NixType>),
+    AttrSet(Vec<Box<NixType>>),


how is this AttrSet type decl supposed to work?

fogti · 2021-06-14T19:34:34Z

+    Str(SmlStr),
+}
+
+trait NixValue {}


why not model all possible values here (including rather opaque stuff like function calls)? Maybe it would be a good idea to put this into a separate crate.

fogti · 2021-06-14T19:35:48Z

-        let kind: rowan::SyntaxKind = NixLanguage::kind_to_raw(NODE_KEY);
+#[derive(Debug, Clone, Eq, PartialEq, Default)]
+pub(crate) struct NixAttrSet<T: Into<GreenNode> + Clone> {
+    kvs: Vec<NixKeyValue<T>>,


why don't we use something like the indexmap crate here?

fogti

necessary because of the removal of the SmlStr newtype

fogti · 2021-11-03T12:35:53Z

@@ -0,0 +1,269 @@
+use crate::parser::utils::{string_to_node, NixNode};
+use crate::SmlStr;


Suggested change

use crate::SmlStr;

use smol_str::SmolStr;

fogti · 2021-11-03T12:36:05Z

+
+#[derive(Debug, Clone, Eq, PartialEq, Default)]
+pub(crate) struct NixKey {
+    pub val: SmlStr,


Suggested change

pub val: SmlStr,

pub val: SmolStr,

fogti · 2021-11-03T12:36:16Z

+
+#[derive(Debug, Clone, Eq, PartialEq, Default)]
+pub(crate) struct NixStringLiteral {
+    pub val: SmlStr,


Suggested change

pub val: SmlStr,

pub val: SmolStr,

fogti · 2021-11-03T12:36:49Z

+//Key(SmlStr),
+//StringLiteral(SmlStr),


Suggested change

//Key(SmlStr),

//StringLiteral(SmlStr),

//Key(SmolStr),

//StringLiteral(SmolStr),

fogti · 2021-11-03T12:37:00Z

+//let mut inputs = Vec::new();
+//if let Some(s) = item.url {
+//inputs.push((
+//SyntaxStructure::Key(SmlStr::new_inline("url")).into(),


Suggested change

//SyntaxStructure::Key(SmlStr::new_inline("url")).into(),

//SyntaxStructure::Key(SmolStr::new_inline("url")).into(),

fogti · 2021-11-03T12:39:30Z

    #[display("{0}")]
+    Bool(bool),
+    #[display("{0}")]
    Other(SmlStr),


??? maybe rebase necessary ???

fogti · 2021-11-03T12:39:48Z

@@ -0,0 +1,21 @@
+use crate::SmlStr;


Suggested change

use crate::SmlStr;

use smol_str::SmolStr;

fogti · 2021-11-03T12:39:56Z

+enum NixPrimitive {
+    Bool(bool),
+    Int(i64),
+    Str(SmlStr),


Suggested change

Str(SmlStr),

Str(SmolStr),

fogti · 2021-11-03T12:40:16Z

+        //let mut inputs_gn = Vec::<Box<dyn>>::new()
+        //if let Some(url) = input.url {
+        //inputs_gn.push((
+        //NixKey{val: SmlStr::new_inline("url")}.into(),


Suggested change

//NixKey{val: SmlStr::new_inline("url")}.into(),

//NixKey{val: SmolStr::new_inline("url")}.into(),

fogti · 2021-11-03T12:40:26Z

+        //let url_gn: GreenNode = url.into();
+        //}
+        //if let Some(is_flake) = input.is_flake {
+        //inputs_gn.push(NixKey{val: SmlStr::new_inline("flake")}.into(), );


Suggested change

//inputs_gn.push(NixKey{val: SmlStr::new_inline("flake")}.into(), );

//inputs_gn.push(NixKey{val: SmolStr::new_inline("flake")}.into(), );

DieracDelta added 3 commits April 26, 2021 23:03

starting out the input generation

ba3f7b3

idiomatic attribute set generation

4b27f8f

updating readme a bit

fbe3676

DieracDelta mentioned this pull request May 2, 2021

Add more options when generating inputs #20

Open

fogti requested changes May 2, 2021

View reviewed changes

Input generation works!

bd9ea68

cleanup

7db4be5

fogti requested changes May 12, 2021

View reviewed changes

more changes

844d87d

fogti requested changes May 16, 2021

View reviewed changes

redoing the rep

e443905

fogti reviewed Jun 14, 2021

View reviewed changes

fogti requested changes Nov 3, 2021

View reviewed changes

	pub fn gen_attr_set(attr_pairs: Vec<(String, String)>) -> GreenNode{
	pub fn gen_attr_set(attr_pairs: &[(String, String)]) -> GreenNode{

		@@ -0,0 +1,269 @@
		use crate::parser::utils::{string_to_node, NixNode};
		use crate::SmlStr;

	//SyntaxStructure::Key(SmlStr::new_inline("url")).into(),
	//SyntaxStructure::Key(SmolStr::new_inline("url")).into(),

	//NixKey{val: SmlStr::new_inline("url")}.into(),
	//NixKey{val: SmolStr::new_inline("url")}.into(),

	//inputs_gn.push(NixKey{val: SmlStr::new_inline("flake")}.into(), );
	//inputs_gn.push(NixKey{val: SmolStr::new_inline("flake")}.into(), );

Conversation

DieracDelta commented May 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DieracDelta May 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

DieracDelta commented May 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DieracDelta commented May 10, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DieracDelta commented Jun 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fogti left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fogti left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

DieracDelta commented May 2, 2021 •

edited

Loading

DieracDelta May 10, 2021 •

edited

Loading

DieracDelta commented May 10, 2021 •

edited

Loading

DieracDelta commented Jun 13, 2021 •

edited

Loading