In traditional WALS models, categorical features are typically represented as one-hot encoded vectors, which can lead to the curse of dimensionality and make it difficult to capture complex relationships between features. Roberta sets, on the other hand, use a learned embedding to represent each categorical feature, allowing the model to capture nuanced relationships between features.